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Abstract 

The  purpose  of  this  research  was  to  develop  a  model  to  convert  existing 
hardcopy  maintenance  manual  fault  Isolation  procedures  into  electronic  format 
and  to  partially  automate  the  fault  isolation  process.  A  comprehensive  model 
was  developed  which  comprised  four  major  components:  user  interface,  expert 
system  submodel,  hypermedia  submodel,  and  submodel  integration  shell.  Using 
this  model,  the  F-15E  nose  landing  gear  fault  isolation  procedures  were 
transformed  into  a  rule-based  expert  system.  An  existing  hypermedia 
information  base  was  expanded  and  subsequently  integrated  with  the  expert 
system  to  form  the  prototype  application. 

The  protot5Tpe  was  evaluated  by  engineering  and  maintenance  personnel 
at  Wright-Patterson  Air  Force  Base.  An  analysis  of  the  survey  responses 
demonstrated  favorable  acceptance  of  the  prototype  and  validated  the 
Integrated  Diagnostic  Expert  System  Model  (IDESM).  Several  conclusions 
from  this  project  and  recommendations  for  further  research  are  presented. 


IX 


INTEGRATED  DIAGNOSTIC  EXPERT  SYSTEM  MODEL  (IDESM): 

A  MODEL  F‘OR  THE  CONVERSION  OF  FAULT  ISOLATION 
PROCEDURES  INTO  AN  INTEGRATED  DIAGNOSTIC  EXPERT  SYSTEM 

I.  Introduction 

The  Australian  Defence  Force  (ADF),  as  well  as  other  western  defense 
forces,  has  been  directed  to  significantly  reduce  manpower  levels.  This 
directive  is  a  result  of  the  Australian  government’s  recent  decision  to  increase 
civilian  contractor  participation  in  support  of  the  ADF,  certain  government 
economic  factors,  and  the  recent  worldwide  trend  towards  peace  at  the 
"superpower"  level.  As  has  been  evident  in  past  reductions,  a  consequence  of 
this  move  towards  a  reduced  force  is  a  reduction  in  the  availability  and 
experience  level  of  workforce  personnel.  Also,  although  the  ADF  is  being 
reduced,  it  is  not  envisaged  that  the  requirement  for  existing  capabilities  will 
be  reduced  proportionately.  Therefore,  all  levels  of  the  force  will  be  required 
to  find  ways  to  improve  their  efficiency. 

This  investigation  addressed  the  benefits  of  developing  a  model  to  convert 
existing  hardcopy  maintenance  manual  fault  isolation  procedures  into 
electronic  format  and  to  partially  automate  the  fault  isolation  process.  The 
flexibility  of  the  model  to  cope  with  other  Service’s  publication  formats  may  be 
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a  topic  for  further  research.  Because  of  their  availability,  United  States  Air 
Force  maintenance  manuals  were  used  in  this  research. 

Background 

The  Royal  Australian  Air  Force  (RAAF)  has  maintained  a  generalist 
training  philosophy  for  its  tradespeople.  That  is,  a  technician  receives  basic 
training  in  general  trade  practices  and  knowledge.  Throughout  Ids/her  career, 
the  technician  tends  to  migrate  from  one  equipment  to  another  and  from  one 
technology  level  to  another,  rarely  specializing  in  one  particular  type  of 
equipment  or  system. 

In  the  past,  this  philosophy  has  worked  because  there  was  considerable 
commonahty  among  systems  and  the  complexity  was  such  that  a  technician 
could  master  many  systems.  However,  with  the  sophistication  of  today’s 
weapon  systems,  the  technician  spends  a  large  proportion  of  his/her  time  in 
training  and  less  time  on  the  job.  The  current  generation  of  weapon  systems 
is  technologically  complex,  with  the  majority  of  subsystems  being  integrated 
and  interactive  in  some  way.  Newer  equipment  is  even  more  complex. 
Although  this  higher  technology  has  led  to  expanded  capability  and  improved 
reliability  of  the  equipment,  it  has  introduced  new  problems. 

The  integrated  nature  of  these  new  systems  malces  it  difficult  to 
specialize  training  to  a  particular  subsystem  in  isolation.  Also,  because  the 
total  system  is  so  large,  it  is  impossible  i.o  fully  train  a  technician  on  the  total 
system  during  the  normal  posting  cycle.  In  the  past  a  technician  achieved  a 
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basic  level  of  competence  through  training,  then  developed  his/her  maintenance 
skills  on  the  job  through  experience  and  repetitive  maintenance  situations, 
which  reinforced  the  learning  process.  The  trend  of  newer  systems  towards 
improved  reliability  and  greater  complexity  has  led  to  a  decrease  in  the 
repetitive  learning  process,  which  has  led  to  reduced  efficiency.  The  technician 
forgets  the  knowledge  gained  from  the  symptoms  and  the  repair  of  a  particular 
fault  before  he/she  again  has  a  chance  to  work  on  a  similar  one. 

A  large  number  of  the  RAAF’s  publications  are  sourced  through  the 
United  States  (US)  Foreign  Military  Sales  (PMS)  program.  As  such,  these 
publications  usually  follow  the  specification  and  format  of  the  parent  weapon 
system’s  technical  orders  (TOs).  Occasionally,  there  are  some  differences  in 
those  publications  where  fleet  specific  changes  are  made,  but  in  the  majority 
oi  cases  these  changes  are  only  in  the  content,  not  the  format.  Therefore,  it 
was  considered  appropriate  to  use  United  States  Air  Force  publications  as  a 
basis  for  this  research. 

The  quantity  of  publications  for  each  new  weapon  system  has  grown  in 
direct  proportion  to  the  increase  in  complexity  of  the  weapon  system.  This 
growth  in  quantity  has  increased  the  cost  of  ownership  for  a  full  set  of  weapon 
system  doexxmentation.  Even  though  the  production  process  has  been  refined 
and  improved,  the  dollar  cost  to  acquire,  reproduce,  bind,  distribute,  and 
maintain  these  publications  is  enormous.  In  addition  to  this  large  monetary 
cost,  the  increase  in  complexity  and  volume  also  leads  to  inefficiencies  in  the 
use  of  a  technician’s  valuable  time  in  two  ways:  time  is  consumed  as  the 
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technician  navigates  the  documentation  during  the  maintenance  process,  and 
technicians,  because  they  are  usually  the  last  link  in  the  chain,  are  often  called 
upon  to  incorporate  the  publication  amendments.  In  both  cases,  the 
technician’s  time  is  diverted  away  from  the  primary  task  of  maintenance. 

On  all  but  the  latest  weapon  systems,  technicians  use  TOs  to  gain 
information  regarding  the  system  tliat  they  are  to  maintain.  When  a 
technician  is  relatively  new  to  a  system,  he/she  tends  to  rely  more  on  the  TO 
(in  particular  the  fault  isolation  procedures)  to  assist  in  the  logical  approach 
to  troubleshooting.  Logical  fault  isolation  relies  on  the  use  of  several  basic 
troubleshooting  rules  and  deduction.  The  TO  fault  isolation  procedures  are  an 
attempt  to  formalize  these  rules  and  reduce  the  amount  of  deduction  required 
by  the  technicians. 

An  expert  system  also  attempts  to  formalize  these  rules  and  automate  the 
fault  isolation  process.  An  expert  system,  which  is  a  software  program  and 
uses  knowledge  collected  from  diagnostic  experts,  is  another  tool  that  the 
technician  can  use  to  help  locate  a  fault  more  accurately  and  with  greater 
speed.  By  making  the  user  interface  simple  to  operate,  minimal  training  is 
required  to  use  the  expert  system.  Although  the  technician  may  be  required 
to  perform  measurements  and  tests  of  the  weapon  system  under  test,  he/she 
usually  only  has  to  respond  to  questions  from  the  expert  system  regarding  the 
state  of  the  system. 

The  introduction  of  expert  systems  throughout  industry  has  led  to  savings 
in  the  technician’s  time  for  training  (less  time  required  to  learn  the  system  and 
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less  time  required  to  develop  deductive  reasoning)  and  more  effective  fault 
isolation  at  a  reduced  experience  level  (18;  1038).  Comparisons  by  Chen 
between  a  human  expert  and  the  use  of  an  expert  system  to  isolate  faults  in 
an  air-conditioner  compressor  resulted  in  the  expert  system  being  superior  in 
both  accuracy  and  speed  (8:342-345).  Therefore,  the  expected  benefits  of 
automation  include  less  training,  more  consistent  quality  of  maintenance,  and 
increased  productivity. 

The  pace  of  information  processing  development  is  such  that  the 
Australian  Defence  Force  expects  that,  with  the  next  major  system  acquisition, 
technical  publications  will  be  provided  in  electronic  format,  rather  than 
hardcopy.  In  his  thesis,  FLTLT  Peter  Cassell  (1991  AFIT  Masters  Graduate) 
developed  a  model  for  the  conversion  of  technical  maintenance  publications 
into  electronic  (hypermedia)  format  (7).  The  Logistics  Research  Division  of  the 
Armstrong  Laboratory  at  Wright-Patterson  AFB  is  upgrading  F-16 
documentation  to  electronic  format  and  is  involved  in  the  development  of  the 
electronic  Integrated  Maintenance  Information  System  (IMIS)  for  the  P-22 
aircraft. 

Several  other  organizations  which  are  detailed  in  Chaptci-  2  are  currently 
developing  diagnostic  expert  systems  that  will  integrate  with  electionic 
(hypermedia)  publications.  However,  there  are  many  existing  fault  isolation 
procedures  which  are  not  likely  to  be  automated,  therefore,  a  model  was 
developed  to  automate  these  existing  fault  isolation  procedures  and  provide  a 
similar  capability  to  that  expected  to  be  delivered  with  future  systems. 
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The  purpose  of  this  research  was  to  develop  a  generic  model  for  the 
conversion  of  maintenance  manual  fault  isolation  procedures  into 
corresponding  expert  systems  and  to  integrate  the  expert  system  structure 


with  a  hypermedia  technical  publication  information  system. 

Research  Obiectives 

The  research  required  to  solve  this  particular  problem  was  divided  into 
several  areas.  Before  a  model  could  be  developed  that  would  convert 
maintenance  manual  fault  isolation  procedures  into  an  expert  system,  the 
input  to  the  model  had  to  be  defined.  The  input  to  the  model  was  developed 
using  information  from  the  United  States  Air  Force  technical  publication 
system.  This  approach  was  valid  because  the  Australian  Defence  Force 
purchases  the  majority  of  its  equipment  from  the  United  States  and  adopts  the 
supplied  equipment  publications  with  little  or  no  alteration  to  the  content  and 
no  alteration  to  the  format. 

Research  Objective  1,  To  determine  the  types  (format/content)  of 
maintenance  manuals  which  the  model  would  convert 

This  objective  involved  a  review  of  the  existing  military  specifications 
which  apply  to  the  various  maintenance  manuals  delivered  to  the  USAF,  and 
attempted  to  determine  common  data  requirements  that  form  the  basis  of 
these  specifications.  Because  the  objective  of  this  research  was  to  develop  a 


generic  model,  It  was  important  to  determine  input  specifications  that  allowed 
the  developed  model  to  be  widely  applicable. 

Research  Objective  2.  To  develop  the  output  specifications  for  the  generic 
model. 

Research  Question  2.1.  How  should  the  expert  system  interface 
with  the  user  to  exchange  fault  isolation  information? 

Research  Objective  2.2.  To  determine  the  structure  of  the  existing 
information  system. 

The  input  and  output  of  the  generic  model  were  defined  with  the 
completion  of  the  first  two  objectives.  The  process  of  conversion  was  then 
identified  and  divided  into  a  sequence  of  steps.  Similar  systems  that  had 
already  been  developed  for  military  and  civilian  applications  were  reviewed, 
and  the  attributes  of  the  various  models  were  assessed  against  this  application. 

Research  Objective  3.  To  select  the  most  appropriate  conversion  model 
methods  and  criteria  from  existing  automatic  fault  isolation  systems. 

Research  Question  3.1.  What  models  currently  exist  that  convert 
fault  isolation  procedures  into  expert  systems? 

Research  Objective  3.2.  To  identify  existing  model  criteria  and 
evaluate  each  criteria  for  suitability  in  this  application. 

Research  Objective  4.  To  develop  a  generic  model. 

Research  Question  4.1.  How  should  the  expert  system  be  structured 
such  that  it  can  be  interfaced  with  existing  systems? 
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Research  Obiective  5.  To  prototype,  test,  and  evaluate  a  sample  system 
with  the  generic  model. 

The  final  stage  of  this  research  was  to  prototj'pe  the  conversion  of  a  fault 
isolation  procedure  based  on  tlie  developed  model  and  to  evaluate  the  model 
for  effectiveness.  Effectiveness  was  defined  as  how  well  the  model  achieved 
the  conversion  objectives. 

Scope/Limitation  of  Research 

Time  was  the  major  limitation  of  this  research.  The  protot5T)e  system 
had  to  be  developed  and  tested  within  a  timeframe  of  six  months.  Because  of 
this  timeframe,  the  research  was  limited  to  developing  a  model  to  convert  fault 
isolation  procedures,  validating  the  model,  and  providing  direction  for  further 
development. 

Definitions 

The  term  "technical  orders"  usually  covers  the  full  range  of  technical 
documentation  for  all  USAF  weapon  systems  and  equipment.  Technical  orders 
differ  according  to  which  specification  was  used  for  their  development.  For 
example,  the  technical  orders  for  the  F-15  aircraft  were  produced  in  accordance 
with  the  Maintenance  Integrated  Data  Access  System  (MIDAS)  developed  by 
the  McDonnell  Aircraft  Company  in  response  to  MIL-M-83495.  MIL-M-83495 
is  the  military  specification  covering  the  Organizational  Maintenance  Manual 
Set. 
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Military  specification  MIL-M-83496A  (13)  requires  that  information  be 
organized  into  a  structured  set  of  technical  manuals,  including  general 
equipment  (GE)  manuals  (referred  to  as  general  vehicle  (GV)  in  TO  1F-15E-2- 
OOGV-00-1  (14)),  general  system  (GS)  manuals,  job  guide  (JG)  manuals,  fault 
isolation  (FI)  and  fault  reporting  (FR)  manuals,  wiring  data  (WD)  manuals, 
schematic  diagram  (SD)  manuals,  combining  manuals,  and  related  manuals 
(13:2-3). 

To  rectify  a  reported  fault,  the  technician  first  needs  to  ascertain  what 
is  causing  the  problem,  i.e.,  isolate  the  fault.  Maintenance  personnel  use  a 
fault  reporting  code  which  can  be  provided  by  the  person  making  out  the  fault 
report  or  derived  from  a  suitable  checkout  procedure.  The  particular  fault 
reporting  code  is  located  in  the  fault  isolation  procedures  contained  in  the  fault 
isolation  manual  for  that  weapon  system  or  equipment.  This  code  indicates 
the  starting  point  of  a  logical  sequence  of  steps  or  tests  that  ultimately  leads 
the  technician  to  the  component  which  most  likely  is  the  cause  of  the  problem. 

The  information  contained  in  these  manuals  can  be  converted  into 
electronic  format,  such  that  the  fault  isolation  procedures  become  an  expert 
system.  The  storage,  manipulation,  and  presentation  of  data  in  this 
management  information  system  (MIS)  is  possible  using  a  hypermedia  system 
to  link  nodes  of  information  which  are  presented  to  the  user  via  a  visual 
display  unit  in  an  associative  user  prescribed  sequence.  Overall,  the  converted 
manuals  become  a  highly  integrated  MIS. 
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Thesis  Overview 

The  management  problem,  research  objectives  and  questions,  the  scope 
of  the  research,  and  definitions  of  terms  were  introduced  in  Chapter  1. 
Chapter  2  contains  a  literature  review  which  was  employed  to  research  the 
field  of  expert  systems  and  the  tools  available  for  expert  system  development. 
This  review  was  also  used  to  scope  the  range  of  specifications  covering  the 
fault  isolation  procedures  covered  in  this  thesis. 

The  methodology  of  the  research  is  covered  in  Chapter  3.  This  chapter 
documents  the  development  of  a  model  for  the  conversion  of  hardcopy  fault 
isolation  procedures  into  an  expert  system  and  the  construction  of  a  prototype 
integrated  expert  system  using  that  model.  Chapter  3  also  describes  the 
testing  methodology  and  evaluation  plan  for  the  prototype.  Chapter  4  presents 
the  survey  findings  and  an  analysis  of  the  results.  Chapter  5  contains  the 
conclusions  drawn  from  the  research  and  provides  recommendations  for  future 
research. 
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II.  Literature  Review 


Introduction 

This  literature  review  summarizes  existing  knowledge  relevant  to  the 
research  problem.  Specifically,  the  aims  of  this  review  are  to  determine  input 
requirements  to  the  model,  identify  any  existing  conversion  models,  and 
examine  current  trends  in  expert  system  development.  Subjects  covered 
include  the  organization  of  technical  manual  fault  isolation  procedures,  the 
concept  of  hypermedia,  the  development  of  information  systems,  a  general 
overview  of  expert  systems  and  models,  and  specific  expert  systems  developed 
to  automate  fault  isolation  procedures.  Sources  accessed  include  the  Defense 
Technical  Information  Center  reports,  the  Aerospace  and  ABI/Inform  on-line 
databases,  and  Engineering  Indexes. 

An  extensive  amount  of  literature  on  expert  systems  exists. 
Cc‘ifee4V\(L.r,tly,  the  literature  search  was  limited  to  erpert  systems  literature 
concerning  expert  systems  development  and  diagnostic,  or  fault  isolation,  types 
of  expert  systems.  The  r<;view  examines  the  approaches  and  tools  used  in 
constructing  expert  systems.  In  addition,  existing  diagnostic  expert  systems 
are  discussed  in  terms  of  approach  adopted,  conversion  models  used,  success 
of  prototype,  and  any  relevant  research  findings.  However,  very  little 
literature  was  found  regarding  models  used  for  converting  existing  fault 
isolation  charts  to  expert  systems. 
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Organization  of  Technical  Manual  Fault  Isolation  Procedures 

In  1976  the  Air  Force  Human  Resources  Laboratory  (AFHRL)  funded  a 

study  for  the  evaluation  of  three  types  of  technical  data  for  troubleshooting 

(25).  From  this  study,  Potter  reported  that  conventional  TOs  of  that  era 

contained  troubleshooting  procedures  that: 

will  generally  isolate  the  malftmction  to  a  particular  stage  and  then  refer 
the  technician  to  the  appropriate  diagram  or  illustration,  'fhe  manuals 
are  prepared  assuming  that  the  technician  has  experience  and  training 
in  the  use  of  test  equipment,  in  locating  most  parts  within  the 
equipment,  and  in  interpreting  schematic  diagrams.  (25:25) 

Logic  Tree  Troubleshooting  Aids  (LTTAs)  were  more  proceduralized  than 

the  TOs  of  that  time  because  they  provided  specific  instructions  both  in  the 

equipment  checkout  procedure  and  in  the  troubleshooting  procedure. 

The  troubleshooting  logic  trees  give  exact  procedures  to  be  performed  and 
ask  questions  about  results  obtained  ...  The  yes  or  no  answer  to  these 
questions  provides  the  logic  in  the  fault  isolation  procedure.  Each  path 
ends  with  instructions  to  replace  a  faulty  component.  (25:25) 

In  comparison,  Fully  Proceduralized  Troubleshooting  Aids  (FPTAs) 

provided  a  further  level  of  troubleshooting  information  in  that  they  gave: 

complete  step-by-step  instructions  for  both  checkout  and  troubleshooting 
of  the  equipment.  These  aids  are  designed  for  use  by  both  experienced 
and  inexperienced  technicians.  A  complete  logical  troubleshooting 
procedure  is  provided  similar  to  the  LTTAs.  Here,  however,  all  specific 
test  equipment  and  tools  are  called  out  in  each  section  as  they  are 
required.  Each  piime  hardware  item  and  the  more  complex  test 
equipments  and  to  is  mentioned  in  the  body  of  the  text  are  accompanied 
by  a  callout  number ...  keyed  to  an  illustration  of  items  appearing  on  that 
same  page.  (25:32) 

The  study  concluded  that  LTTAs  and  FPTAs  were  superior  to  the  existing 
technical  orders.  As  a  result,  military  specifications  now  include  the  LTTA 
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format  for  fault  isolation  procedures.  Military  Specification  (MIL-M-83496A) 
(13)  is  the  latest  specification  detailing  the  content  and  format  of  data  which 
is  required  to  be  included  in  the  organizational  maintenance  manual  set.  The 
requirements  for  the  FI  manual  are  included  in  Section  3.5  of  this 
specification.  The  quantity  of  current  USAF  TOs  which  conform  to  the  LTTA 
format  as  specified  in  military  specifications  since  1976  (including  MIL-M- 
83495A)  could  not  easily  be  ascertained.  However,  more  manuals  comply  to 
the  LTTA  format  than  to  MIL-M-83495A.  Therefore,  the  LTTA  format  became 
the  basis  for  developing  the  input  specification  for  the  Integrated  Diagnostic 
Expert  System  Model  (IDESM).  The  major  benefit  of  selecting  the  LTTA 
format  over  a  particular  military  specification  is  that  the  model’s  applicability 
is  increased. 

Hypermedia  Definition 

Hypertext  and  hypermedia  are  synonymous  terms.  Although  there  is  no 
generally  accepted  definition  for  these  systems,  they  are  characterized  by 
several  features  (2:820).  The  information  stored  within  the  hypermedia  system 
is  chunked  into  small  units  variously  called  notecards,  frames,  or  nodes.  Each 
individual  unit  may  contain  text,  graphics,  video,  spreadsheets,  sound,  or 
animation.  The  information  is  displayed  one  unit  at  a  time,  with  the  units  of 
information  being  interconnected  by  links.  These  links  are  displayed  to  the 
user  to  enable  navigation  through  the  hypermedia  database  for  the  required 
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information.  Figure  1  gives  examples  of  the  four  different  methods  for 
structuring  hypermedia  information. 


Hierarchical  Links  Keyword  Links 


Referential  Links  Cluster  Links 


Figure  1.  Types  of  Hypermedia  Links  (27:36) 


Hierarchial  linking  establishes  a  structure  in  the  information  base  that 
uses  a  menu  to  link  to  the  next  block  of  data  to  be  accessed.  Ke3nvord  linking 
identifies  words  that  are  embedded  in  the  text  as  links  to  associated  data. 
Usually  the  linked  data  does  not  provide  access  to  further  information. 
Referential  linking,  the  preferred  form  of  h3rpermedia,  provides  links  from 
virtually  any  unit  of  information  to  all  associated  units.  All  of  the  information 
on  a  subject  matter  is  completely  interconnected.  Cluster  linking  provides  less 
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capability  than  referential  linking  because  it  is  usually  limited  to  linking 
within  a  small  cluster  of  imits. 

The  Development  of  Management  Information  Systems 

Nolan  has  categorized  the  development  of  management  information 
systems  within  an  organization  into  six  stages  (23:115-126).  The  first  stage  of 
system  development  occurs  when  several  low-level  operating  systems  are 
introduced  into  functional  areas  such  as  accounting  and  inventory  control. 
When  other  fimctional  areas  see  the  success  of  these  initial  systems  there  is 
a  rapid  expansion  of  computing  applications,  which  results  in  the  second  stage. 
The  third  stage  of  information  system  development  occurs  when  management 
attempts  to  control  the  vaiious  computing  resources.  This  stage  is 
characterized  by  formalized  planning,  with  the  emphasis  on  rebuilding  effective 
systems,  professionalizing  existing  applications,  and  improving  documentation. 
During  the  fourth  stage  previous  applications  are  upgraded  using  database 
management  systems.  At  this  stage  there  is  a  change  in  orientation  from 
management  of  individual  computer  assets  to  management  of  the  company’s 
data  resources.  In  the  fifth  stage,  data  administration  is  introduced  and  the 
various  applications  are  integrated.  The  final  stage  of  information  system 
growth  occurs  when  all  of  the  company’s  applications  are  integrated,  and  the 
system  structure  reflects  both  the  organization  structure  and  the  information 
flows  within  the  company. 


Based  on  these  stages  of  growth,  the  majority  of  information  systems 
within  the  Australian  Defence  Force  can  be  categorized  as  being  in  either 
stages  four  or  five.  However,  the  current  emphasis  is  on  developing  integrated 
systems.  The  expert-  system  developed  during  this  study  is  an  information 
system  that  is  capable  of  integration  with  existing  information  systems  to 
improve  and  expand  the  overall  information  management. 

Expert.  System  Definition  and  Structure 

An  expert  system  may  be  defined  as  a  software  application  that 
represents  the  knowledge  and  judgement  of  a  person  who  is  a  recognized 
expert  in  a  particular  field  and  uses  this  knowledge  to  solve  problems  or  mimic 
human  reasoning  within  that  domain  (17:131). 

According  to  Waterman  (30:22-24),  the  general  structure  of  an  expert 
system  includes  a  knowledge  base,  an  inference  engine,  and  a  user  interface 
(see  Figure  2).  The  knowledge  base  contains  the  formal  representation  of  all 
of  the  knowledge  provided  by  the  expert(s).  This  base  is  termed  the  domain 
knowledge.  The  inference  engine  is  a  program  responsible  for  interpreting  the 
contents  of  the  knowledge  base  in  relation  to  a  user-specified  input  or 
hypothesis  in  order  to  determine  a  goal  or  conclusion.  The  user  interface  is  the 
mechanism  which  allows  the  user  to  communicate  with  the  expert  system. 

The  inference  engine  contains  specific  information  on  the  current  state 
of  the  problem/solution  referred  to  as  the  context.  This  element  can  provide 
an  explanation  facility  that  informs  the  line  of  reasoning  to  the  user.  Because 
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the  process  of  drawing  inferences  from  a  knowledge  base  is  essentially  the 
same  for  similar  problems,  a  single  inference  engine  can  function  wHith  a 
variety  of  expert  systems  manipulating  different  knowledge  bases.  Inference 
engines  are  usually  available  as  part  of  the  higher-level  language  or  within  the 
shell  of  an  expert  system  development  tool. 


Knowledge  Representation 

The  most  common  methods  of  representing  knowledge  are  rules,  facts  and 
objects,  semantic  networks,  and  frames  (3:29). 

Rule-Based  Expert  Systems.  In  a  rule-based  expert  system,  the  domain 
knowledge  is  represented  as  a  set  of  rules  (IF  condition(6)  exist,  THEN  action) 
that  is  checked  against  a  collection  of  facts  or  knowledge  known  about  the 
current  situation  (4:26).  Each  rule  is  given  a  certainty  factor  (usually  a  value 
between  0  and  1)  which  expresses  the  degree  of  confidence  placed  in  the  rule. 
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Deductive  and  Inductive  Reasoning.  The  methodology  for  deri\4ng 
expert  system  rules  can  be  classified  according  to  the  two  main  types  of 
scientific  reasoning:  deductive  and  inductive  reasoning.  Deductive  rules  are 
developed  for  expert  systems  when  the  expert  can  explain  the  logic  behind  the 
reasoning  process  in  terms  of  which  attributes  of  the  problem  were  considered 
and  the  relative  importance  of  each  attribute.  However,  a  problem  arises  in 
developing  rules  for  the  expert  system  when  the  expert  cannot  logically  explain 
how  or  why  a  particular  decision  was  made.  This  situation  occurs  because  the 
expert  uses  the  inductive  reasoning  process  and  his  own  experience  and 
knowledge  to  make  an  inferential  jump  from  the  facts  of  the  problem  to  the 
solution.  Because  of  the  difficulty  that  experts  have  in  quantifying  the 
inductive  reasoning  process,  several  systems  have  been  developed  from  which 
valid  and  predictive  rules  are  derived  from  a  list  of  expert  decisions.  One  of 
the  most  popular  methods  uses  the  Interactive  Dichotomiser  Mark  3  (IDS) 
algorithm.  The  IDS  algorithm,  developed  by  J.  Ross  Quinlan,  is  a  computer 
program  designed  to  induce  the  most  efficient  set  of  rules  from  a  set  of 
example  cases  (22:379-383),  This  computer  program  analyzes  all  of  the 
supplied  decisions  and  attempts  to  develop  a  rule  hierarchy  that  explains  the 
decision  making  process. 

Inference  Mechanisms.  In  a  rule-based  expert  system,  there  are 
sevei'al  methods  of  inference.  The  two  most  common  methods  include 
backward  chaining  and  forward  chaining  (4:28-34).  In  backward  chaining,  the 
inference  mechanism  works  "backward"  from  the  THEN  part  of  the  rule  to  the 
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IF  part  of  the  rule.  The  inference  mechanism  guesses  at  the  conclusion.  It 
then  attempts  to  prove  that  its  guess  is  correct  by  locating  a  rule  whose  THEN 
condition  matches  tlie  guess  and  by  establishing  that  all  conditions  contained 
in  the  IP  part  of  that  rule  exist.  In  forward  chaining,  the  inference  mechanism 
first  establishes  facts  and  then  .deduces  new  facts  by  working  forward  through 
tlic  rules  (from  the  IF  part  to  the  'CHEN  part).  The  decision  to  use  forw’ard 
chainiTig  or  backward  chaining  is  noimally  based  on  the  relative  quantities  of 
facts  and  conclusions  in  the  rule  base.  Forward  chaining  is  more  appropriate 
if  a  problem  involves  significantly  more  conclusions  than  facts.  Backward 
chaining  is  better  suited  to  problems  that  involve  significantly  more  facts  than 
conclusions. 

Blackboard  Architecture.  Blackboard  architecture  is  used  when 
either  the  domain  knowledge  or  the  control  aspects  of  the  problem  become  too 
complicated  to  be  contained  within  a  simple  rule  base  (17:131-136). 
Consequently,  independent  groups  of  rules  called  knowledge  sources  are 
established,  and  the  groups  communicate  through  a  central  database  called  a 
blackboard  (see  Figure  3).  The  knowledge  provided  by  each  group  consists 
primarily  of  intermediate  results  cf  problem  solving.  'iTie  structure  of  this 
system  normally  includes  a  monitoring  expert  system  that  continuously 
examines  the  input  data  and  proddes  its  interpretatioixs  to  other  expert 
systems  which  perform  specific  functions  such  as  diagnosis  or  control.  Because 
this  architecture  allows  each  system  access  to  central  data,  the  system  can  be 
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developed  incrementally,  with  each  expext  system  being  validated 
independently. 


Figure  3.  Data  Representation  of  Blackboard  Architecture 


Fact-Based  Expert  Systems.  The  knowledge  base  for  an  expert  system 
may  also  contain  facts  about  the  problem  domain.  Facts  may  be  descriptions 
of  objects  and  relationships  between  objects  in  the  domain  under  consideration. 
By  using  the  technique  of  propositional  calculus  anti  the  knowledge  of  the 
existiiig  facts,  the  system  can  then  determine  v.'hether  assertions  about  objects, 
concepts,  or  events  are  true  or  false  (24:46-50). 
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Semantic  Network  Expert  Systems.  A  semantic  network  is  a  method  for 


representing  the  natural  relationships  that  occur  among  objects.  A  semantic 
network  is  described  by  points  (nodes)  connected  by  a  series  of  directed  links 
(arcs  or  arrows)  that  show  the  liierarchical  relations  between  two  nodes. 
Nodes  represent  objects,  concepts,  or  events.  Arcs  can  be  defined  in  a  variety 
of  ways  depending  on  the  type  of  knowledge  that  they  represent.  Pigford 
states  that: 

One  of  the  main  advantages  of  the  use  of  semantic  nets  is  that  almost 
any  object,  attribute  (characteristic),  or  concept  can  be  defined  and 
relationships  created.  But  there  is  a  trade-off  for  this  flexibility.  There 
are  no  standard  guidelines  for  forming  semantic  nets,  so  their  form  can 
vary  from  system  to  system.  In  contrast,  prediction  rules  have  a 
standard  format.  (24:43) 

Frame-Based  Expert  Systems.  Frame-based  expert  systems  use  a 
knowledge  representation  method  that  associates  features  with  nodes 
representing  concepts  or  objects.  The  nodes  are  connected  into  a  network 
hierarchy  by  using  links  to  describe  the  relationships  between  nodes.  The 
topmost  nodes  represent  general  concepts  and  the  lower  nodes  more  specific 
instances.  Nodes  lower  in  the  hierarchy  automatically  inherit  properties  of 
higher-level  nodes.  This  method  provides  a  natural,  efficient  way  to  categorize 
and  structure  a  taxonomy  such  as  medical  diseases  or  speech  recognition 
(30:73-79).  The  features  of  each  node  are  described  in  terms  of  several 
attributes  (slots)  and  their  values.  The  idea  is  to  represent  data  in  a  frame 
system  with  constraints  between  sets  of  values.  The  process  of  adding  or 
removing  values  from  the  slots  can  violate  the  constraints  and  activate 


procedures  attached  to  the  slots.  These  procedures  may  then  modify  values  in 
other  slots,  continuing  the  process  until  the  desired  goal  is  achieved  (30:390). 

Generic  Categories  of  Expert  System  Applications 

The  generic  categories  of  expert  systems  are  listed  in  Table  1.  Further 
review  has  been  limited  to  expert  systems  that  have  been  developed  in  the 
diagnosis  category. 

TABLE  1 

GENERIC  CATEGORIES  OF  EXPERT  SYSTEMS  (30:33) 


Problem  Addressed 


Interpretation 

Prediction 

Diagnosis 

Design 

Planning 

Monitoring 

Debugging 

Repair 

Instruction 

Control 


Inferring  situation  descriptions  from  sensor  data 
Inferring  likely  consequences  of  given  situations 
Inferring  system  malfunctions  from  observables 
Configuring  objects  under  constraints 
Designing  actions 

Comparing  observations  to  expected  outcomes 
Prescribing  remedies  for  malfunctions 
Executing  plans  to  administer  prescribed  remedies 
Diagnosing,  debugging,  and  repairing  student  behavior 
Governing  overall  system  behavior 


Problem  Characteristics  that  Require  Expert  Systems 

To  require  an  expert  system  solution,  certain  characteristics  should  exist 
in  a  maintenance  fault  isolation  or  diagnosis  problem.  First,  the  problem 
should  be  sufficiently  complex  to  need  an  expert  system,  but  fiie  scope  has  to 
be  narrow  enough  to  allow  all  of  the  knowledge  in  the  domain  to  be  defined. 
Second,  expertise  in  solving  the  problem  must  be  scarce  within  the 
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organization,  but  the  problem  must  be  capable  of  being  solved  by  an  expert 
within  a  few  hours.  Third,  at  least  one  recognized  expert,  who  is  willing  to  act 
as  a  source  of  information,  must  be  available  and  able  to  adequately  express 
his  reasoning.  Fourth,  there  must  be  a  logical  process  involved  in  diagnosing 
the  problem  that  does  not  require  the  novice  to  use  large  amounts  of  intuition. 
Finally,  there  must  be  a  large  benefit  in  resolving  the  problem  quickly  (6:459- 
460). 

Choosing  a  Tool  for  Building  Expert  Systems 

There  are  two  software  alternatives  for  developing  expert  systems  -  an 
expert  system  shell  software  package  or  a  programmiing  language.  A  variety 
of  software  shells  are  commercially  available.  Individual  products  vary 
according  to  the  type  of  knowledge  representation  they  support,  the  inference 
strategies  employed,  user  and  external  interfaces,  knowledge  base  size  and 
performance  limitations,  machine  availability,  and  cost  (30:142-149). 

The  typical  expert  system  shell  is  marketed  as  a  package  consisting  of  a 
compiler  to  translate  the  rule  base  from  a  software  language  into  an  internal 
representation  and  a  run-time  system  to  apply  the  compiled  knowledge  base 
to  the  user’s  problem.  Shell  systems  are  the  most  straightforward  means  of 
rapidly  prototyping  an  expert  system,  as  the  developer  does  not  have  to  provide 
the  inference  strategy  or  run-time  support.  Programming  languages  offer  more 
flexibility  than  shells,  but  they  also  require  the  developer  to  design  the 
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knowledge  base  and  implement  the  inference  engine  that  accesses  the 
knowledge  (5:80-81). 

Existing  Fault  Isolation  Expert  Systems 

Several  diagnostic  expert  systems  that  have  been  prototyped  are  now 
reviewed  and  their  research  findings  presented.  The  methodologies  which 
were  used  for  representing  knowledge  include  rules  and  frames.  The  systems 
discussed  are  a  representative  sample  of  the  types  of  fault  isolation  expert 
systems  that  have  been  developed  and  reflect  the  developer’s  preference  for 
rule-based  systems. 

Fired  Heater  Advisor.  One  example  of  the  application  of  blackboard 
architecture  is  the  Fired  Heater  Advisor,  which  is  used  to  control  a  fired  heater 
in  the  chemical  industry.  "A  fired  heater  is  a  complex  piece  of  equipment 
whose  control  demands  close  monitoring  of  the  process  parameters  and  fine 
judgement  in  reacting  to  subtle  changes  in  them"  (17:136).  The  Fired  Heater 
Advisor  monitors  such  parameters  as  temperature,  fuel-gas  composition,  and 
control  valves.  The  expert  system  continually  checks  the  parameter  values 
and,  if  any  abnormal  condition  is  detected,  automatically  invokes  a  diagnostic 
system.  The  expert  system  determines  the  pertinent  rule  base  relevant  to  the 
problem,  performs  the  diagnosis,  and  notifies  the  process  controller  of  the 
conclusions. 

An  Expert  Database  System  for  Shipboard  Maintenance.  Lieutenant 
Commander  Van  Hook  (29)  developed  a  prototype  for  troubleshooting  the  NAXI 
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100-2  low  pressure  air  compressor  system.  The  objective  of  his  research  was 
to  combine  an  expert  system  shell  with  a  database  management  system.  The 
troubleshooting  guide  from  the  technical  manual  was  translated  into  a  series 
of  if-then  rules  for  the  expert  system.  The  expert  system  identified  a  fault  and 
then  accessed  a  separate  database  to  provide  detailed  procedures  to  correct  the 
problem.  The  fact  that  the  database  for  this  prototype  was  limited  to  254 
character  fields  proved  to  be  a  major  limitation.  As  a  result,  detailed 
maintenance  procedures  could  not  be  accessed,  and  graphics  could  not  be 
displayed. 

Interactive  Fault  Diagnosis  and  Isolation  System  (IFDIS).  Several 
versions  of  IFDIS  have  been  developed  for  diagnosis  of  failures  in  jet  aircraft 
engines.  These  include  fault  diagnosis  of  the  TF-30  engine  in  the  F-111  aircraft 
(19:15-16)  and  fault  diagnosis  of  the  F404  jet  engine  in  the  F/A-18  aircraft  (20). 
The  main  aim  of  establishing  these  expert  systems  was  to  convert  a  large 
volume  of  technical  documentation  into  a  more  accessible  format  and  allow 
appropriate  levels  of  access  for  different  experience  levels  of  technicians.  The 
F404  IFDIS  was  established  using  the  E]\fVCIN  shell.  EMYCIN  is  a  backward 
chaining  rule-based  application  package  that  is  most  suited  to  tasks  that  can 
be  expressed  as  structured  selection  problems.  Structured  selection  occurs 
when  eacii  successive  question  asked  by  the  expert  system  reduces  the  list  of 
failures  that  could  have  resulted  in  the  observed  problem.  Solutions  that  are 
not  excluded  by  the  user’s  answers  are  presented  at  the  end  of  the  consultation 
in  order  of  suspected  causation.  The  knowledge  base  for  IFDIS  was  created  by 
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converting  the  binary  fault  tree  tables  in  the  work  package  documentation  into 
rules.  Even  though  the  prototype  was  successful,  major  conclusions  from  the 
research  concerned  the  following:  the  poor  performance  of  the  expert  system 
shell,  a  desire  to  express  the  knowledge  base  in  a  symptom-conclusion 
structure,  and  adequacy  of  the  initial  work  package  documentation. 

Automotive  Computer-based  Expert  (ACE)  (21:189-191).  ACE,  developed 
through  the  Artificial  Intelligence  Laboratorj'  at  the  University  of  Alabama,  is 
a  diagnostic  expert  system  for  the  Altec  D-900  Derrick  vehicle  operated  by  the 
Alabama  Power  Company.  ACE’s  knowledge  representation  schemes,  inference 
mechanisms,  and  explanation  capability  are  all  written  in  the  progi'amming 
language  Microsoft  The  system  contains  approximately  8000  lines  of  code 
and  required  one  man-year  of  programming  effort.  The  knov/iedge  base 
contains  approximately  200  IF-THEN  rules  and  uses  a  conventional  backward 
chaining  inference  mechanism.  The  unique  feature  of  ACE  is  the  graphic 
support  provided  to  the  troubleshooting  program.  This  capability  is  made 
possible  by  the  integration  of  digitized  and  edited  35mm  photographs  in  the 
ACE  knowledge  base.  Images  are  provided  by  the  system  in  both  mandatory 
(safety  aspects,  special  procedures)  and  optional  modes  (user  can  request  video 
information). 

Jet  Engine  Technical  Advisor  (JETA)  (1:154-157).  JETA  was  developed 
using  the  strategy  of  viewing  the  diagnostic  knowledge  as  a  hierarchy  of  fault 
nodes.  Each  node  represents  a  single  fault  or  problem  frame  in  the  knowledge 
base.  The  hierarchy  models  the  diagnostic  knowledge  from  its  most  general 
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to  its  most  specific  faults.  Thus,  the  engine  is  considered  to  have  several 
general  faults  (e.g.,  acceleration).  Each  of  these  faults  becomes  a  branch  in  the 
hierarchy  and  is  specialized  until  its  descendent  fault  nodes  point  to  specific 
faulty  subsystems  in  the  engine.  Each  rule  frame  provides  the  reasoning 
algorithm  for  alternatives  to  the  current  node  or  to  its  descendants  given 
certain  observations.  At  the  time  of  publication  of  the  article,  there  were 
approximately  120  frames  in  the  knowledge  structure,  and  the  expert  system 
was  being  field  tested  and  evaluated  by  potential  users. 

Integration  of  Naw  Stock  Points  Expert  Systems  (26:1-34).  Students 
from  the  Naval  Postgraduate  School  had  previously  developed  three  different 
expert  systems  to  aid  inventory  managers  at  Navy  Stock  Points.  The  objective 
of  Lieutenant  Rouska's  research  was  to  convert  these  stand-alone  systems  into 
an  integrated  expert  system.  A  major  portion  of  Rouska’s  efforts  involved  the 
conversion  of  the  existing  knowledge  bases,  which  were  written  in  the 
programming  leinguages  Ml^^  and  PBOLOG^"^,  into  VP-Expert^'^  rule  bases. 
A  hierarchical  system  architecture  was  developed  that  used  an  integration 
module  to  control  which  of  the  three  different  systems  were  to  be  accessed. 
The  prototype  was  successful  and  demonstrated  that  it  is  possible  to  integrate 
several  different  knowledge  bases. 

Summary 

Although  a  plethora  of  information  on  experts  systems  exists,  this 
literature  review  only  covered  the  subject  areas  relevant  to  the  research 
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problem.  The  basic  types  of  expert  system  were  introduced,  and  the  different 
methodologies  for  representing  knowledge  explained.  The  review  of  existing 
expert  systems  was  restricted  to  systems  that  are  within  the  diagnosis 
category.  Because  little  information  regarding  models  used  to  convert 
diagnostic  procedures  into  an  expert  system  knowledge  base  exists,  the 
knowledge  gained  from  previous  work  in  the  field  was  used  to  develop  the 
outline  for  the  generic  model  to  be  developed  in  the  rest  of  the  research  task. 
This  knowledge  primarily  included  the  organization  of  fault  isolation 
procedures  and  the  requirements  of  the  relevant  military  specifications  and 
was  discussed  at  the  beginning  of  the  chapter.  Finally,  because  the  research 
objective  for  integrating  this  expert  system  with  other  information  systems  is 
part  of  an  overall  strategy  to  improve  the  development  of  information  systems 
for  the  ADF,  the  stages  in  information  system  development  were  briefly 
reviewed. 
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III.  Methodology 


Overview 

This  chapter  describes  the  methodology  used  to  meet  the  objectives 
outlined  in  Chapter  1.  The  methodology  includes  an  explanation  of  the 
research  design,  the  processes  used  in  developing  the  input  and  output 
specifications  for  the  model,  and  an  outline  of  the  plan  for  model  development. 
Existing  knowledge  determined  through  the  literature  review  was  aligned  with 
the  methodology. 

Each  research  objective  and  its  associated  findings  are  reported.  The 
model  is  then  presented  with  an  explanation  of  each  element  of  the  model. 
The  chapter  describes  how  a  prototjrpe  system  was  developed  from  the  model 
and  details  the  stages  of  testing  and  evaluating  the  prototype.  The  chapter 
also  outlines  the  development  of  the  survey  instrument,  the  data  collection 
plan,  and  the  method  of  analysis. 

Explanation  of  Method  and  Research  Design 

There  are  typically  five  stages  in  the  development  of  an  expert  system 
(30:139-140).  The  first  stage,  or  demonstration  prototype,  is  used  to  solve  a 
portion  of  the  problem  being  undertaken.  This  first  stage  is  used  to  verify  the 
problem  definition,  scope  the  task,  determine  adequacy  of  domain  knowledge 
representation,  and  determine  suitability  of  expert  systems  technology.  Stage 
two  occurs  when  the  system  Is  expanded  into  a  research  prototype  which 
displays  credible  performance  on  the  entire  problem.  These  systems  tend  to 
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be  fragile  because  they  lack  sufficient  testing  and  revision  to  meet  all  possible 
situations. 

In  the  third  stage,  the  system  evolves  into  a  field  prototype  which 
contains  a  medium-to-large-sized  program  that  has  been  revised  through 
extensive  testing  on  real  problems  in  the  field  environment.  These  systems 
have  good  performance  with  adequate  reliability  and  user-friendly  interfaces. 
The  final  stages  of  the  production  model  and  commercial  system  occur  after  the 
prototype  systems  have  been  extensively  field-tested,  obtained  the  necessary 
quality  and  performance  criteria  to  meet  the  user’s  requirements,  and  are 
suitable  for  production.  A  typical  system  can  take  five  years  to  be  fully 
developed.  Due  to  the  time  limitations  imposed  on  this  research  project,  the 
primary  objective  was  tc  develop  a  prototype  which  combined  the  elements  of 
stages  one  and  two. 

Concept  of  Use 

The  methodology  utilized  for  the  model  design  was  based  on  a  realistic 
concept  of  information  usage  within  the  next  decade.  The  technical 
environment  is  foreseen  as  being  almost  paperfree.  All  of  the  information 
required  by  a  technician  to  perform  any  task  on  any  piece  of  equipment  within 
the  entire  defense  inventory  could  be  stored  and  maintained  in  various  on-line 
databases.  A  technician,  tasked  to  repair  an  aircraft  fault  for  example,  would 
log  on  to  a  centralized  computer  and  enter  the  details  of  the  system  that 
requires  repair.  The  computer  would  download  the  required  information  to  a 
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portable  computer  that  the  teclmician  would  then  take  out  to  the  aircraft  to 
perform  the  maintenance  task. 

The  technology  to  perform  this  level  of  informalioxi  Dicnagement  is  biing 
developed  and  should  be  available  within  the  next  five  years  ,  Several  civilian 
companies  that  manufacture  notepad  systems  were  contacted,  with  the 
intention  of  demonstrating  die  prototype  on  a  notepad  system  to  different 
organizations  at  Wright-Patterson  Air  Force  Base.  NCR  company  provided,  on 
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long-term  loan,  an  NCR  3125  NotePad  system.  This  NotePad  is 
approximately  A4  size  (9.8"  x  11.7"  x  1"),  weighs  3.95  poimdc,  uses  Microsoft 
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MSDOS  and  Wiiidows  for  Pen  Computing  as  operating  systems,  and 
utilizes  a  passive  pen,  instead  of  a  computer  keyboard,  for  input. 

Model  Design  Methodology 

In  this  research  the  following  steps  were  utilized  in  developing  the 
IDESM: 

a.  Discussions  with  local  techmcal  publication  authoritiec  working  for  the 
Air  Force  Material  Command  (AFMC)  indicated  that  the  latest  military 
specification  covering  the  requirements  for  Organizational  Maintenance 
Manual  Sets  (OMMS)  was  MIL-M-83495A.  In  the  time  available,  it  was 
impossible  to  identify  every  specification  or  format  that  has  been  used  in 
procurement/development  of  existing  USAF  publications  and  which 
publications  conform  to  the  LTTA  format.  However,  the  technical 
publication  staff  at  AFMC  confirmed  that  the  majority  did  conform  to  the 


31 


LTTA  format.  Therefore,  the  model  input  specification  was  deliberately 
kept  as  broad  as  possible  to  cover  the  greatest  range  of  existing  hardcopy 
manuals. 

b.  Current  USAF  technical  publication  specifications  and  technical 
maintenance  publications  for  the  F-15E  aircraft  were  analyzed  to 
determine  the  basic  structure  of  the  knowledge  used  for  fault  isolation 
and  the  structure  of  any  associated  data. 

c.  Cassell’s  Structured  Hypermedia  Application  Development  Model 
(SHADM)  proto  ype  was  dissected  and  analyzed  to  determine  the  basic 
format  of  the  hyp-'^rtext  document,  the  linking  arrangements,  and  any 
interface  peculiarities. 

d.  Local  experts  were  consulted  regarding  the  impact  of  Computer-aided 
Acquisition  and  Logistics  Support  (CALS)  and  the  direction  other 
integrated  applications  such  as  IMIS  were  taking. 

e.  Local  engineering  and  maintenance  staT  were  consulted  with  regard  to 
user  interface  requirements. 

The  results  of  these  steps  are  outlined  below  in  the  next  section.  However,  the 
intention  of  this  approach  was  to  provide  the  framework  on  which  to  build  the 
model.  The  framework  ser/ed  as  the  basis  for  determining  the  knowledge 
representation  methodology  and  the  format  of  necessary  text  files,  indices,  and 
the  user  interface.  It  also  provided  ihe  strategies  for  integrating  the  expert 
system,  the  hypermedia  system,  and  the  user  interface. 
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This  section  presents  the  results  of  the  research  into  each  of  the 
objectives  and  discusses  the  impact  of  those  results  on  the  model  design.  The 
first  research  objective  was  to  determine  the  types  (format/'content)  of 
maintenance  manuals  which  the  model  will  convert.  The  objective  in  designing 
a  generic  model  is  to  ensure  that  the  model  has  wide  application.  Therefore, 
instead  of  selecting  certain  military  specification(s)  and  possibly  limiting  the 
model’s  applicability  to  manuals  that  comply  with  those  specifications,  the 
decision  was  made  to  use  the  underlying  concepts  behind  the  development  of 
these  military  specifications  as  the  baseline.  The  literature  review  determined 
that  most  military  specifications  since  1976  have  been  developed  using  Logic 
Tree  Troubleshooting  Aids  (LTTAs)  in  the  fault  isolation  procedure. 

LTfAs  provide  exact  procedures  to  be  followed  during  the  equipment 
checkout  and  troubleshooting  proceduicb  and  iise  Yes/No  questions  to  provide 
the  logic  to  ieclate  the  fault.  These  Yes/No  questions  are  ideal  for  conversion 
to  knowledge  base  rules  for  an  expert  system.  The  minimum  input 
requirement  for  the  Integrated  Diagnostic  Expert  System  Model  was  therefore 
that  the  technical  manual(s)  being  converted  to  an  electronic  form  have  LTTAs 
(or  an  equivalent  method  of  data  representation). 

The  second  research  objective  was  to  develop  the  output  specifications  for 
the  generic  model.  The  output  specification  for  the  model  was  based  on  the 
concept  of  using  a  portable  maintenance  aid  (PMA).  The  designers  believe  that 
the  developed  system  must  be  capable  of  being  used  by  the  technician  at  the 


actual  maintenance  site,  which  means  that  the  system  must  be  suitable  for  use 
on  an  aircraft,  in  hangars/workshops,  oii  a  flightline,  or  on  deployment.  The 
designers’  intent  was  to  determine  generic  output  specifications  that  could  be 
refined  during  subsequent  protot)rping.  As  the  system  was  designed  as  a  PMA, 
the  output  specification  consists  primarily  of  the  requirements  for  the  format 
and  display  characteristics  of  the  PMA  screen  output.  Because  a  printer  is  not 
part  of  the  foreseen  system  configuration,  the  format  of  any  printed  output  was 
not  pursued  further. 

In  designing  output  screens,  areas  are  required  for  headings  and  titles, 
the  content  of  the  display,  and  messages  and  instructions  (27:408).  Figure  4 
shows  the  screen  presentation  format  adopted  for  IDESM.  Headings  and  titles 
are  displayed  at  the  top  of  the  screen  to  inform  the  technician  of  his  current 
position  within  the  system.  Messages  and  instructions  are  displayed  at  the 
bottom  of  the  screen.  The  screen  display  area  in  the  middle  may  be  used  for 
the  display  of  text  or  graphic  information  or  both.  A  split  screen  enables  the 
text  describing  the  fault  isolation  procedure  to  be  displayed  on  the  left  side  of 
the  screen.  Any  additional  information,  such  as  graphics,  can  be  displayed  on 
the  right  side  of  the  screen. 

Two  special  features,  scrolling  and  maximizing,  are  required  to  enhance 
the  readability  of  the  information  displayed  on  the  output  screen.  The  detailed 
irdbrmation  provided  in  many  hardcopy  graphics  (e.g.,  schematic  and  wiring 
diagrams  that  are  presented  on  several  foldout  pages)  is  unable  to  be  displayed 
on  one  display  screen.  Therefore,  the  user  must  have  the  capability  to  move 


34 


(scroll)  aro^ind  the  screen  to  the  required  part  of  the  graphic  and,  if  required, 
be  able  to  maximize  the  graphic  to  provide  a  fullscreen  view.  Scrolling  or 
paging  can  also  be  used  if  the  user  wishes  to  access  a  segment  of  text  which 
exceeds  one  screen  length.  Therefore,  the  capability  to  scroll  vertically  and 
horizontally  and  the  ability  to  maximize  the  graphics  (transform  to  full  screen 
view)  were  provided  by  the  IDESM.  These  capabilities  are  consistent  with  the 
requirements  of  the  draft  military  specification  MIL-M-GCSFUI  (9). 
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Figure  4.  IDESM  Screen  Presentation  Format 


In  addition,  research  objective  two  questioned  the  interfacing 
requirements  of  the  user  with  the  expert  system.  The  PMA  is  likely  to  have 
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either  a  limited  keyboard  or  rely  on  input  from  the  user  via  a  pointing  device 
(pen).  The  user  interface  should  therefore  be  designed  to  minimize  the  data 
entry  requirements  from  the  technician.  The  user  should  be  able  to  choose  any 
function  or  option  by  selecting  it  from  a  displayed  list.  Because  the  system  will 
be  used  by  technicians  with  varying  experience  levels,  the  "system  should  be 
able  to  lead  the  newcomer  by  the  hand  without  burdening  the  experienced 
user"  (10:97).  The  user  interface  should  also  be  capable  of  providing  on-line 
assistance  and  be  designed  to  minimize  the  chance  of  user  error.  The 
commands  and  instructions  displayed  to  the  user  should  be  in  everyday 
English  and  not  use  any  specialized  computer  terminology.  The  method  used 
for  presentation  of  text  and  graphical  information  should  be  consistent 
throughout  the  information  base  and  should  conform  to  any  normal  technical 
stereotypes. 

Research  objective  two  also  addressed  the  structure  of  the  existing 
information  system  which  was  developed  during  the  design  and  testing  phases 
of  the  Structured  Hypermedia  Application  Development  Model  (7).  This  system 
was  based  on  CALS  requirements  as  detailed  in  MIL-STD-1840A  (11:11).  The 
seven-bit  ASCII-formatted  information  was  stored  in  separate  files  by  a  vehicle 
system.  Although  the  hypermedia  links  were  created  using  the  software 
package  Hyperwriter  ,  minor  modification  to  the  linking  information  within 
the  ASCII  files  allowed  access  to  the  data  by  other  hypermedia  systems  and, 
in  particiiiar,  the  software  shell  used  for  development  of  the  IDESM  prototype. 
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The  third  research  objective  was  to  select  the  most  appropriate 
conversion  model  methods  and  criteria  from  existing  automatic  fault  isolation 
systems.  The  literature  review  discussed  several  existing  expert  systems  and 
the  various  knowledge  representation  methodologies  used  for  their 
development.  The  review  essentially  identified  that  there  were  four  methods 
for  representing  knowledge  within  an  expert  system:  production  rules,  facts 
and  objects,  semantic  networks,  and  frames.  The  use  of  production  rules  is  the 
most  common  method  for  representing  knowledge,  accounting  for  over  ninety 
percent  of  developed  expert  systems.  The  reason  for  the  popularity  of  this 
method  is  the  ease  of  developing  the  associated  logic  and  the  large  number  of 
commercially  available  software  packages  that  support  rule-based  systems. 

An  initial  requirement  of  objective  three  was  to  determine  the  existence 
of  any  models  that  currently  convert  fault  isolation  procedures  into  expert 
systems.  However,  the  literature  did  not  reveal  any  models  specifically 
developed  for  the  conversion  of  fault  isolation  procedures  into  expert  systems. 
The  literature  review  revealed  the  following: 

a.  The  major  criterion  for  converting  existing  fault  isolation  procedures  into 
an  expert  system  consists  of  structuring  the  resulting  knowledge  base 
according  to  the  requirements  (syntax)  specified  by  the  software  shell  or 
programming  language. 

b.  General  guidelines  for  organizing  rules  or  frames  within  the  knowledge 
base  do  not  exist. 
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In  summary,  the  expert  systems  that  have  been  previously  developed  have 
been  constructed  according  to  the  programming  language  or  software  shell 
used. 

A  further  research  objective  was  to  identify  existing  model  criteria  and 
evaluate  each  criterion  for  suitability  in  this  application.  As  no  models  were 
found  that  specifically  converted  fault  isolation  procedures  into  an  expert 
system  structure,  the  authors  decided  to  use  the  principles  employed  in 
structured  programming  and  software  design  for  development  of  the  model. 

Principles  of  Soaware  Design  (27:696-70?) 

A  structured  system  is  one  that  is  developed  from  the  top  down  and 
divided  into  manageable  components  or  modules.  Senn’s  principles  of  software 
design  include: 

Modularity  and  Partitioning.  The  design  of  the  system  should  consist  of 
a  hierarchy  of  modules,  with  each  module  performing  a  specific  function. 
Lower-level  modules  are  generally  smaller  in  scope  and  size  compared  to 
higher-level  modules  and  serve  to  partition  processes  into  separate  functions. 

Coupling.  Coupling  refers  to  the  strength  of  relations  between  modules. 
Modules  should  be  constructed  with  minimal  dependence  on  other  modules  in 
the  system. 

Cohesion.  Cohesion  refers  to  the  strength  of  relations  within  a  module. 
Modules  should  contain  all  of  the  elements  required  to  perform  a  single 
processing  fimction. 
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Span  of  Control.  Span  of  control  lefers  to  the  number  of  subordinate 
modules  controlled  by  the  calling  module.  Modules  should  interact  with  and 
manage  the  functions  of  a  limited  number  of  lower-level  modules. 

Size.  The  number  of  instructions  contained  in  a  module  should  be  limited 
so  that  module  size  is  generally  small. 

Shared  Use.  Fimctions  should  not  be  duplicated  in  separate  modules  but 
should  be  established  in  a  single  module  that  can  be  used  by  any  other  module 
when  needed. 

This  structured  system  methodology  was  chosen  primarily  because  of  the 
advantages  gained  fi’om  using  the  concepts  of  modularity  and  shared  use. 
Modularity  allows  the  system  to  be  developed  in  stages.  More  importantly,  any 
required  maintenance  during  the  life  of  the  system  can  be  controlled  at  the 
module  level  without  requiring  a  complete  rewrite  of  the  software.  Also, 
modules  at  the  same  hierarchical  level  can  be  designed  according  to  a  common 
stnicture,  thus  producing  a  more  controlled  design  and  thereby  increasing 
maintainability. 

Shared  use  of  modules  provides  several  additional  advantages,  including 
reducing  the  amount  of  software  that  must  be  designed  and  written, 
minimizing  the  number  of  changes  required  during  system  maintenance,  and 
decreasing  the  chance  of  software  errors.  With  shared  use  of  modules, 
information  normally  duplicated  throughout  hardcopy  maintenance  manuals 
(e.g.,  the  safety  warning  that  precedes  every  maintenance  action)  needs  to  be 
stored  only  once  in  an  electronic  system  and  the  appropriate  connections 


39 


established  for  its  shared  use.  This  approach  reduces  the  amount  of  storage 
space  required  for  aircraft  documentation. 

Generic  Model  Develoument 

The  Integrated  Diagnostic  Expert  System  Model  (IDESM),  developed  to 
fulfill  the  requirements  of  research  objective  four,  contains  four  major  elements 
'  the  user  interface,  the  hypermedia  submodel,  the  expert  system  submodel, 
and  the  submodel  integration  shell  (see  Figure  5).  The  research  findings  from 
the  first  three  objectives  were  incorporated  into  the  design  of  the  model. 

The  major  design  considerations  of  the  user  interface  have  been  detailed 
previously  in  this  chapter  under  research  objective  2.  Therefore,  the  design 
considerations  of  the  hypermedia  submodel,  the  expert  system  submodel,  and 
the  submodel  integration  shell  will  be  presented  in  the  following  sections. 

Development  of  the  Hypermedia  Submodel 

The  implementation  of  hypermedia  within  the  model  requires  the 
definition  of  several  control  strategies  -  storage,  chunking,  and  linking. 

Storage.  Discussions  with  several  experts  on  technical  documentation 
suggested  that  the  storage  requirements  for  an  entire  set  of  aircraft 
publications  on  electronic  media  would  require  between  700  and  1000 
Megabytes  of  storage.  Because  large  amounts  of  data  are  being  accessed,  the 
processing  time  required  to  locate  and  extract  the  required  unit  of  information 
from  storage  must  be  minimized.  The  major  element  of  processing  time  is  the 
time  required  to  access  a  hard  disk.  Therefore,  the  stored  data  must  be 
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structured  such  that  a  minimal  number  of  disk  accesses  are  required  to  extract 
the  required  data.  Because  of  the  slow  response  time  generated,  a  sequential 
read  of  a  database  of  this  size  searching  for  the  required  data  would  produce 
an  unusable  system.  Therefore,  the  most  efficient  method  for  data  storage  is 
to  use  an  index  file  as  a  pointer  to  the  actual  location  of  the  required  data. 
Extraction  of  the  required  data  then  requires  a  read  of  the  index  file  and  direct 
access  to  the  data. 


Figure  5.  IDES  Components 

Chunking.  The  information  stored  in  each  unit  or  node  should  be 
segmented  into  logical  chunks  and  should  not  be  limited  to  the  amount  of  data 
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that  can  be  displayed  on  one  computer  screen.  If  thii  unit  of  information 
exceeds  one  computer  screen,  then  the  user  should  be  provided  with  the  ability 
to  access  the  entire  unit  of  information. 

Linking.  Linking  is  the  process  whereby  units  of  information  are 
connected  together  by  a  software  pointer.  Referential  linking,  as  described  in 
Chapter  2,  should  be  implemented  within  the  hypermedia  information  base, 
and  an  unlimited  number  of  links  should  be  possible  from  any  one  unit  of 
information.  There  are  two  software  methods  for  implementing  links  within 
hypermedia  documents.  The  first  method  requires  the  developer  to  identify 
both  ends  of  the  link  and  the  actual  interconnection  path.  Using  this  method, 
the  developer  must  re-establish  the  link  every  time  the  information  within  the 
database  is  changed.  This  method  is  obviously  unsuitable  for  large  databases 
with  dynamic  data.  The  second  approach  is  to  create  a  virtual  link  using  a 
keyword  or  phrase.  In  virtual  linking,  a  keyword  (e.g.,  safety)  identifies  a  link 
10  further  information  on  safety  procedures.  The  indexed  database  is  then 
searched  to  locate  the  keyv^ord  and  its  linking  information  and  a  virtual 
pointer  to  the  relevant  information  is  established.  Because  the  link  is  only 
established  as  needed  and  maintenance  of  only  the  keywords  and  the  index  is 
required,  this  approach  is  quite  flexible. 

Hypermedia  Submodel  Text.  The  structure  of  the  hypermedia  submodel 
is  illustrated  in  Figure  6.  The  logical  hierarchical  format  of  the  OMMS 
produces  the  follomng  major  groupings  of  information:  aircraft,  system, 
subsystem,  sub-subsystem,  and  line  replaceable  unit  (LRU).  The  system, 
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subsystem,  sub-subsystem  groups  can  be  further  subdivided  into  general 
(description/overview)  or  specific  (principles  of  operation,  special  maintenance 
requirements,  test  equipment,  etc.)  information.  This  subdivision  generates 
eight  types  of  text  information,  each  of  which  requires  different  linking 
capabilities. 


Hypermedia  information  Base 


T«rt 

Graphics 

Audlo/Vkleo 

•  Aircraft  G«n«riBl 

-  General  Aircraft 

Views 

(Future) 

-  Descriptive 

•  Syttam  Qsnerel 

-  System  Specific 

•  •  System  Position 
Overlays 

-  Prooedurol 

•  Sub  System  General 

-  Sub  System  Spedfle 

•  Detailed  Dtegrams 

•  Sub  Sub  System 

•  Schematic  Diagrams 

General 

•  Sub  Sub  System 

•  Special  Tool 

Specific 

Illustrations 

-Une  Replaoeable 

-  Photographic  Quailly 

Unit  Specific 

(Color  or  Shades  of 
Gray)  Illustrations 

Figure  6.  Composition  of  Hypermedia  Submodel 


The  GV  manual  contains  general  aircraft  and  system  general  information 

which  should  be  available  for  global  access  within  the  hypermedia  information 

base.  Information  within  the  GV  manual  includes: 

a  short  description  of  each  system,  general  maintenance  procedure 
information;  danger  areas  and  precautions;  dimensions  and  areas; 
information  relating  to  lifting  and  jacking,  levelling,  towing,  parking  and 
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mooring;  and  information  relating  to  aircraft  servicing.  Consolidated  lists 
of  supplies,  test  equipment^  and  special  tools,  complete  with  part 
numbers  ...  all  abbreviatioi>R  and  electrical,  electronic,  and  mechanical 
symbols.  (14:1-1) 

The  information  describing  the  lower  six  levels  is  distributed  throughout  the 
general  system  manuals,  the  fault  isolation  manuals,  and  the  job  guides. 
System  specific  and  subsystem  general  information  is  contained  in  the  General 
System  manuals.  More  detailed  information,  regarding  the  subsystem  specific, 
sub-subsystem  (general  and  specific),  and  LRU  specific  can  be  found  in  the  GS 
manuals,  FI  manuals,  and  JGs.  This  information  should  be  available  for 
access  depending  on  the  task  being  performed  and  the  needs  of  the  user. 

Hypermedia  Submodel  Grauhics.  Five  different  types  of  graphic 
information  have  been  identified. 

General  Aircraft  Views.  These  illustrations  aro  simplified  graphic 
drawings  of  the  aircraft  tliat  identify  locations  of  subsystems/LRUs.  Typically, 
the  same  foundation  drawing  is  reproduced  with  an  overlayed  identifying 
pointer  and  text  each  time  a  different  location  is  illustrated.  To  minimize  the 
storage  of  graphic  information  in  electronic  format,  the  designers  propose  the 
use  of  ct*mmon  aircraft  base  drawings  and  specific  overlays  to  identify  the 
required  item.  This  approach  is  consistent  with  paragraph  3.3.3.2  of  (draft) 


MII^M-GCSFUI  (9). 

Detailed  Diaerrams.  These  diagrams  provide  the  user  with  a 
detailed  illuEtration  of  the  subsystem/LRIIs  supported  by  numbered  call-outs 
to  a  tabular  component  list. 


44 


Schematic  Diagrams.  These  diagrams  provide  specific  electrical  and 
hydraulic  schematics  for  the  subsystem/sub-subsystem/LRUs. 

Special  Tool  Illustrations.  These  illustrations  provide  information 
on  special  tools  that  are  required  for  task  performance. 

Photographic  Quality  Illustrations.  Photographic  quality 
illustrations  (black  and  white  or  color)  are  photographs  that  can  be  utilized  to 
provide  information  that  cannot  be  displayed  adequately  in  drawings. 
Photographs  are  expensive  and  difficult  to  reproduce  in  hardcopy  publications 
at  the  resolution  needed  to  transfer  information.  However,  for  use  in  a 
hypermedia  application,  the  photographs  can  be  inexpensively  digitized  and 
stored  electronically.  Once  digitized  and  stored,  the  photographs  can  be 
manipulated  the  same  as  any  other  digital  data.  The  only  limitations 
associated  with  the  hypermedia  use  of  photographic  quality  data  are  the 
memory  needed  to  store  and  manipulate  the  data  and  the  speed  penalties 
incurred  by  the  ipplication  in  processing  the  larger  amounts  of  data  . 

Hypermedia  Submodel  AudioA^ideo.  The  h3q)erriiedia  model  includes  the 
capability  for  integrating  audio/video  into  the  system.  The  two  types  of 
audio/video  presentations  identified  are: 

Descriptive.  Descriptive  audio/video  presentations  can  be  used 
instead  of  text  for  several  topics,  including  principles  of  operation  and  system 
description. 


45 


Procedural.  Procedural  audio/video  presentation  can  be  used  to 
guide  the  technician  through  general  maintenance  and  fault  isolation 
procedures. 

The  information  stored  within  the  model’s  knowledge  base  should 

conform  to  several  other  standards.  The  u.se  of  standardized 

Systero/Sub83rstein/Subject  Number  (S/S/SN)  assignment  throughout  the 

technical  manuals,  is  in  accordance  with  MIL-STD-1808,  provides  maximum' 

cross  referencing,  and  reduces  the  search  time  required  to  locale  data. 

The  information  stored  in  electronic  ibrmat  should  comply  with  any  CAIjS 

requirements.  CALS  specifications,  which  include  MIL-M>2800iA,  MIL-M* 

28002,  and  MIL-M-28003,  establish: 

the  requirements  for  the  digital  data  form  of  page  orientated  technical 
publications.  Data  prepared  in  conformance  to  these  requirements  will 
facilitate  the  automated  storage,  retrieval,  interchange,  and  processing 
of  technical  documents  from  heterogeneous  data  sources.  (12:1) 

The  format  for  data  storage  within  the  model  should  therefore  follow  the 

guidelines  associated  with  the  use  of  the  Standard  Generalized  Markup 

Language  (SGML). 

fn  summary,  the  hypermedia  submodel  has  partitioned  the  data  within 
the  information  base  into  text,  graphics,  and  audio/video.  Each  of  these  areas 
has  then  been  further  partitioned  into  functional  and  logical  access  areas.  The 
submodel  has  also  considered  the  methods  of  storage,  chunking,  and  linking 
the  necessarjr  information  to  facilitate  an  integrated  system. 
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Reseaixh  objective  four  also  questioned  how  the  expert  system  should  be 
stiuctured  so  that  it  could  be  interfaced  with  existing  systems.  Because  the 
LTTA  structure  was  suitable  for  direct  conversion  into  a  series  of  IF/THEN 
rules,  the  method  of  knowledge  representation  chosen  for  the  expert,  system 
was  rules.  The  estimation  of  the  number  of  rules  that  would  be  required  in  a 
knowledge  base  for  all  the  fault  isolation  procedures  associated  with  a 
particular  aircraft  is  extremely  difficult.  The  designers  believe  that  literally 
thousands  of  rules  would  be  necessary.  Consequently,  a  modular  structure  was 
developed,  as  shown  in  Figure  7.  This  structure  follows  the  principles  of  top- 
down  design  and  was  based  on  the  standardized  S/S/SN  assignment.  The 
highest  level  of  knowledge  base  which  relates  to  the  aircraft  or  general  vehicle 
(e.g.,  F15E)  was  designed  as  a  control  module  that  would  allow  the  user  the 
option  of  either  selecting  a  lower  level  system  for  fault  analysis  or  directly 
accessing  the  hypermedia  inforaiaticn  base. 

The  next  level  which  relates  to  the  aircraft  system  (e.g.,  landing  gear) 
v/as  also  designed  as  a  control  module.  The  user  should  be  able  to  either  select 
a  subsystem  for  fault  analysis  or  access  the  hypermedia  information  base  at 
that  level  or  a  liigher  level.  The  knowledge  base  at  the  subsystem  level  (e.g., 
extension  and  retraction  system)  includes  the  rule  base  for  the  fault  analysis 
checkout  procedure.  This  module  provides  guidance  to  the  teclmiclan  through 
the  checkout  procedure  and  links  directly  to  the  relevant  faullcode  information 
when  a  failure  is  identified  at  a  procedural  step. 
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Fiffurci  7.  Expert  Sj'stem  Submodel 


The  knowledge  base  at  the  faultcode  level  (e.g.,  3230N1ZZ)  comprises  the 
specific  rules  associated  with  the  identified  fault  and  assists  the  technician  to 
identify  the  faulty  comporie>it.  On  identification  of  the  faulty  component,  the 
system  should  be  able  to  autc<matically  load  and  run  the  associated 
removal  installation  procedures.  If,  however,  the  technician  already  knows  the 
faultcode  or  realizes  that  a  particular  item  needs  to  be  replaced,  he/she  should 
be  able  to  select  the  ’"rlevant  procedure  from  the  higher-level  menu  without 
first  having  to  access  a  checkout  procedure. 
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The  submodel  integration  shell  must  be  able  to  integrate  with  both  the 
user  interface,  the  expert  system,  and  the  hypermedia  submodels.  The  shell 
must  be  able  to  provide  indexed  access  to  the  hypermedia  information,  the 
necessary  inference  mechanism  for  execution  of  the  expert  system,  and  the 
layout  requirements  of  the  user  interface  (including  a  scroll  capability  for  units 
of  information  that  exceed  one  display  screen  and  a  maximize  capability  for 
graphics).  This  integration  shell  should  be  either  written  in  a  programming 
language  by  experienced  programmers  or  purchased  off-the-shelf. 

l?iat9t.YPfi.D?V9lQpm^nt 

The  final  research  objective  was  to  prototype,  test,  and  evaluate  sample 
systems  with  the  generic  model.  To  test  his  model,  FLTLT  Cassell  (7:57) 
developed  a  hypermedia  prototype  based  on  the  F-15E  Nose  Landing  Gear 
(NLG)  system.  Approximately  50  percent  of  the  GV  manual  was  converted  to 
hypertext.  Because  one  of  the  objectives  of  this  design  was  to  integrate  the 
prototyped  expert  system  with  an  existing  hypermedia  system,  the  prototype 
of  the  IDESM  was  based  on  the  F-15E  NLG  system. 

Because  of  the  time  limitation  imposed  on  development  of  the  prototype, 
the  designers  were  tmable  to  develop  their  own  integration  shell  and  purchased 
an  off-the-shelf  system.  Several  off-the-shelf  expert  system  shells  were 
evaluated  for  their  hypermedia  capabilities  and  applicability  to  the  windows 
operating  environment.  Knowledge  Pro  for  Windows  (KPW)  was  chosen  as 


the  software  for  use  in  the  prototype.  However,  this  selection  should  not  be 
seen  as  a  recommendation  for  the  use  of  this  package  over  any  other  product. 

The  concept  behind  the  prototype  development  is  illustrated  in  Figure  8. 
The  complete  text  of  the  extension  and  retraction  system  fault  analysis 
checkout  procedure  (15:30-11  30-17)  was  converted  into  a  subsystem-level 

knowledge  base  (refer  to  Figure  7).  This  checkout  procedure  guides  the 
technician  to  a  faultcode  in  the  range  of  3230N1ZZ  to  3230NYZZ  (15:30-75  - 
30-171).  These  32  faultcodes  were  converted  into  modular  faultcode-level 
knowledge  bases  which  can  be  called  by  the  "checkout  procedure"  expert 
system.  Each  of  these  modules  was  designed  using  a  standard  template  which 
is  included  in  Appendix  A.  Graphics  scanning  difficulties  prevented  the 
authors  from  providing  a  full  set  of  graphics  to  accompany  the  checkout  and 
faultcode  procedures. 

Two  control  modules,  representing  the  aircraft  and  system  level 
knowledge  bases,  were  developed  to  demonstrate  the  integration  of  the  expert 
system  with  the  hypermedia  publication.  A  considerable  amount  of  the  textual 
information  called  by  the  hypermedia  links  was  based  on  FLTLT  Cassell’s 
SHADM.  Overall,  approximately  1  megabyte  of  ASCII  and  graphics  files  were 
produced  representing  a  broad  range  of  different  textual  and  graphics 
information  that  pertains  to  the  F-15E  aircraft  and  F-15E  NLG  system. 

An  example  of  the  text  and  index  files  created  using  the  hypennedia 
submodel  and  KPW  are  also  presented  in  Appendix  A.  The  basic  text  file 
stmeture  contains  ASCII  text  and  uses  the  delimiters  //  to  divide  the 
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informaticn  into  usable  chunks.  The  character  string  between  the  delimiters 
and  the  fix’st  end  of  line  character  define  the  search  keyword.  KPW  provides 
an  indexing  capability  to  speed  up  access  to  the  chunks  of  information.  An 
index  file,  required  for  each  text  hie,  contains  the  keyword,  the  start  address, 
and  the  length  of  each  chunk. 


Figure  8.  Developed  PrototjTpe 


To  establish  its  hypermedia  links,  KPW  requires  that  the  character  s  #m 
be  placed  around  the  keyword,  e.g.,  SAFETY  becomes  #mSAFETY#m.  These 
hypermedia  links  are  underlined  on  the  screen  and  displayed  in  a  different 
color  or  font  to  provide  the  user  with  a  visual  indication  that  there  is  more 


information  available  on  that  topic.  KFW  lets  the  developer  set  the  color  and 
font  of  the  hypertext  links  and  therefore  permits  the  use  of  different  colors  to 
signify  different  applications.  For  example,  a  "safety"  hypertext  link  may  be 
displayed  in  red,  graphics  may  be  displayed  in  blue,  and  other  ordinary  text 
Units  may  be  displayed  in  green.  Several  other  control  characters,  such  as  #f 
and  #b,  allow  the  developer  to  select  the  color  of  the  foreground  and 
background  for  individual  on-screen  special  effects.  However,  care  must  be 
taken  not  to  overwhelm  the  use?^  witli  garish  effects. 

£YMMigI)L£la.n 

The  two  factors  involved  in  selecting  the  test  environment  were  proximity 
cf  testing  personnel  and  the  availability  of  a  suitable  hypermedia  publication 
information  system.  The  preferences  were  to  find  a  maintenance  section  with 
the  appropriate  personnel  that  was  prepared  to  test  the  prototype  and  was 
located  at  or  close  to  Wright-Patterson  Air  Force  Base.  Such  a  situation  would 
afford  the  researchers  better  access  to  the  participants  and  simplify  the  survey 
procedures. 

Engineers  and  maintenance  personnel  from  the  F-15E  System  Program 
Office  (SPO),  C-17  SPO,  4960th  Test  Wing,  Armstrong  Laboratories,  and  local 
RAAF  maintenance  commimity  were  approached  to  evaluate  the  research 
prototype.  Where  possible,  the  prototype  was  compared  with  existing  hardcopy 
teclmical  manuals.  A  survey  of  the  engineers  and  maintenance  personnel 
involved  in  the  evaluation  was  conducted  regardiiAg  the  following  criteria; 
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a.  ease  of  use  (user  being  able  to  do  what  he  or  she  wants  to  do  easily), 

b.  suitability  of  user  interface  (screen  layout,  buttons,  colors,  pointer,  and 
scroll/maximize  functions), 

c.  user  level  adequacy  (experienced  versus  inexperienced), 

d.  correctness  of  converted  procedure, 

e.  completeness  of  converted  procedure, 

f.  efficiency  (effect  on  time  to  make  serviceable), 

g.  reliability  and  consistency  of  results,  and 

h.  overall  usefulness  of  the  system. 

Because  the  prototype  was  only  to  be  a  stage  one/two  prototype,  the  main 
thrust  of  the  evaluation  phase  was  to  verify  that  the  IDESM  was  adequate  and 
that  the  technique  used  for  domain  knowledge  representation  was  suitable. 
Two  additional  purposes  of  evaluating  the  prototype  were  to  determine  if  the 
goals  of  a  reduction  in  tiaining  commitment  (evidenced  by  the  ability  of  low- 
experienced  users  being  able  to  effectively  fault  find  with  the  aid  of  the 
prototype)  and  the  integration  of  the  information  systems  were  effective.  The 
results  of  the  survey  were  used  to  evaluate  the  prototype  and  suggest  areas  of 
further  development  for  the  model. 

Description  of  Population  and  Sample 

In  this  particular  study,  the  relevant  population  was  all  possible  users  of 
integrated  expert  systems  that  could  be  developed  using  this  particular  model. 
The  sampling  process  conformed  to  the  non-probability  judgement  sample 
design  (16:275).  The  sample  of  interest  was  technical  and  engineering 
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personal  of  the  DSAP  and  RAAF  who  were  based  locally  and  available  to 
evaluate  the  prototype.  The  size  of  the  sample  was  to  be  in  the  range  of  fifteen 
to  twenty-five  participants. 


ImtmmeM  DevelQgmenj;_  and  Testing 

A  survey  instrument,  which  is  included  in  Appendix  B,  was  developed  for 
evaluation  of  the  prototype.  Content  and  sampling  validity  of  the  survey 
instrument  were  confirmed  with  tlie  assistance  of  Air  Force  Institute  of 
Techriology  staff.  External  validity  of  the  survey  was  based  on  the  premise 
that  tlie  survey  instrument  was  to  be  applied  to  a  representative  sample  cf 
United  States  and  Australian  Air  Foices’  engineers  and  technicians. 

Data  Collection  Plan 

The  survey  was  conducted  by  questionnaire.  The  majority  of  questions 
in  the  survey  instrument  required  respondents  to  express  an  opiiiion  on  the 
performance  of  the  research  prototype  voth  respect  to  a  particular 
measurement  criterion.  The  sun^ey  also  provided  respondents  with  the 
opportunity  to  furnish  ei  ther  general  or  specific  comments  on  each  of  the  topics 
covered  in  the  questionnaire.  For  the  opinion  questions,  respondents  were 
asked  to  score  each  peTformance  criterion  on  a  six  point  Likert  scale  (16:220). 
The  scale  ranged  from  strongly  disagree  (a  score  of  1)  to  strongly  agree  (a  score 
of  6).  The  intervals  between  each  score  value  were  assumed  to  be  equal,  thus 
providing  interval-scaled  data  for  analysis. 


Interval-scaled  data  can  be  statistically  summarized  by  using  an 
arithmetic  mean  as  the  measure  of  central  tendency  and  a  standard  deviation 
as  the  measure  of  dispersion.  The  main  statistic  of  interest  in  this  analysis 
was  the  arithmetic  mean.  On  a  Likert  scale  of  one  to  six,  the  middle  score  of 
3.6  relating  to  a  respondent’s  opinion  of  undecided  was  therefore  discouraged. 
For  the  expert  system  prototype  to  be  an  improvement  over  the  current 
hardcopy  fault  isolation  manuals,  the  survey  results  should  provide  a  mean 
that  is  greater  than  3.5.  The  data  from  the  survey  was  analyzed  using  the 
statistical  package,  Statistix^  Version  3.5.  The  purpose  of  the  analysis  was  to 
determine  the  central  tendency  of  the  data  collected  for  each  survey  question 
and  then  determine  which  measures  of  the  model  scored  in  the  categories  of 
mean  being  less  than  3.6,  equal  to  3.5,  and  greater  than  3.5.  The  significance 
of  the  collected  data  was  then  analyzed  with  respect  to  the  original  research 
objectives.  The  analysis  is  presented  in  Chapter  4. 

Summary 

This  chapter  outlined  the  methodology  used  to  solve  the  research  problem 
stated  in  Chapter  1.  The  reasoning  behind  using  a  prototype  approach  to  the 
problem  was  explained.  The  existing  knowledge  in  the  research  area,  revealed 
by  the  literature  review,  was  summarized  and  used  as  a  basis  in  planning  the 
methodology  behind  the  model  and  prototype  design.  A  plan  for  evaluating  the 
model  and  the  prototype  in  the  field  and  for  surveying  the  participants  was 


outlined.  Finally,  the  basis  for  the  statistical  analysis  of  the  survey  results 
was  explained. 
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Overview 


The  evaluation  plan,  as  presented  in  Chapter  3,  identified  eight  areas  for 
measurement  of  the  prototype  performance.  This  chapter  subdivides  the  30 
questions  used  in  the  survey  instrument  into  those  eight  areas,  analyzes  the 
responses  to  each  survey  question,  and  then  draws  conclusions  on  the 
performance  of  the  prototype.  Any  significant  comments  made  by  the  survey 
respondents  are  also  discussed.  Finally,  the  overall  performance  of  the 
prototype  is  assessed,  and  the  IDESM  is  evaluated. 

Survey  Results 

A  total  of  20  surveys  were  received  from  local  engineering  and 
maintenance  personnel.  The  survey  results  are  presented  in  Appendix  C. 
Included  are  the  comments  made  by  the  respondents,  the  statistical 
measurements  of  mean,  standard  deviation  (SD),  95%  confidence  interval 
(96%CI),  and  the  niunber  of  respondents  (n)  for  each  particular  question. 
Questions  22,  23,  and  24  produced  a  low  survey  response  rate  because  many 
of  the  survey  respondents  did  not  liave  access  to  the  necessary  F-15E 
puldications. 

The  sample  of  survey  respondents  included  four  members  from  the  F-15E 
SPO,  two  members  from  the  C-17  SPO,  nine  members  from  the  4950th  Test 
^^  ing,  two  members  from  Armstrong  Laboratories,  and  three  local  RAAF 
representatives.  The  rank  level  of  the  survey  respondents  included  six  officers, 
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eight  non-commissioned  officers  (technical  sergeants),  and  four  airmen.  This 
sample  was  repre  sentative  of  the  available  population.  Therefore,  the  results 
of  the  survey  can  be  used  to  draw  conclusions  on  the  effectiveness  of  the 
prototype. 

The  survey  was  originally  planned  to  be  conducted  using  the  NCR  3125 
Notepad  computer.  However,  because  thei%  were  some  importation  difficulties, 
the  notepad  was  not  available  when  required  and  the  survey  was  conducted 
using  IBM  personal  computers.  The  notepad  has  since  been  used  to 
demonstrate  the  prototype  to  several  organizations  at  Wright-Patterson  Air 
Force  Base.  Infoimal  feedback  from  the  demonstrations  has  indicated  that 
higher  scores  may  have  resulted  if  the  survey  was  conducted  using  the 
Notepad  computer  instead  of  personal  computers. 

TJio  survey  used  a  six  point  Likert  scale  ranging  from  Strongly  Disagree 
(a  score  of  1)  to  Strongly  Agree  (a  score  of  6),  A  mean  score  of  3.5  would  reflect 
neither  agreement  or  disagreement  (a  result  of  indifference)  with  the 
statement  being  surveyed.  The  statistical  analysis  of  the  survey  resulted  in 
a  mean  of  greater  than  3.5  for  all  of  the  questions  (statements).  Furthermore, 
for  26  of  the  30  questions,  the  entire  96%CI  for  the  mean  was  above  3.5.  This 
result  allows  the  authors  to  state  with  confidence  that  the  population  would 
agi'ee  with  the  respective  statements.  Although  questions  4,  20,  22.  and  24 
had  the  lower  limit  of  the  95%CI  below  3.5,  these  survey  results  are  not 
discussed  separately.  Each  question  is  addressed  within  its  respective 
performance  assessment  area. 
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Ease  of  Use 


’Ease  of  use’  is  an  ill-deiined  term.  Therefore,  for  the  purposes  of  this 
research,  T.ase  of  use’  was  defined  as  ’the  ability  of  the  user  to  easily  do  what 
they  want  to  do.’  The  following  five  questions  were  used  to  assess  ease  of  use: 
Ql.  The  program  is  easy  to  start.  (Mean  =  5.38) 

Q2.  The  program  is  easy  to  use.  (Mean  =  4.49) 

Q3.  Navigating  throughout  the  fault  analysis  system  is  easy.  (Mean  =  4.18) 

Q4.  Sufficient  explanations  (on-screen/documentation)  are  provided  to  use  the 
system.  (Mean  =  4.08,  95%CI  3.46  -  4.70) 

Q5.  Exiting  the  system  is  easy.  (Mean  =  4.82) 

The  prototype  scored  highly  in  all  five  questions,  with  only  questioki  four  not 
having  the  entire  95%CI  above  3.5.  Discussion  of  significant  comments 
received  in  this  area  follows. 

Several  conunents  suggested  that  more  guidance  could  have  been 
provided  to  the  user  to  further  improve  the  ease  of  use  of  the  system. 
Suggestions  included  more  instructional  messages  advising  the  user  to  ’Press 
continue  for  next  step’  or  advise  whether  a  single  or  double  click  of  the 
pointing  device  is  required  to  make  a  selection.  The  requiremerkt  for  single  or 
double  clicks  was  explained  in  the  on-screen  help  function  proidded  with  the 
system.  However,  the  feedback  suggests  that  the  average  user  does  not  use 
on-screen  help  (except  as  a  last  resort)  and  seldom  reads  instructions  before 
using  the  program.  The  requirement  for  standardizing  to  single  or  double 
clicks  (events)  is  debatable.  (Does  one  standardize  for  simplicity  or  use 
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separate  events  to  differentiate  between  distinct  functions?)  This  topic  is  a 
candidate  for  further  research. 

The  combination  of  single/double  clicks  and  lack  of  oiii-screen  instructions 
were  poor  design  features  of  the  prototype  The  prototype  designers  did 
attempt  to  standardize  to  one  click  because  the  soi^ware  documentation  and 
supplier  indicated  it  was  possible.  However,  as  a  result  of  their  inexperience 
with  the  software  package,  the  designers  were  unable  to  accomplish 
standardization.  The  protob'pe  can  definitely  be  improved  in  terms  of 
providing  more  guidance  to  the  user.  However,  an  important  goal  of  the 
designer  is  to  provide  sufficient  guidance  to  the  novice  without  slowing  down 
the  expert.  Future  versions  of  the  prototype  should  be  easier  to  use. 

Several  users  commented  on  the  loss  of  visibility  of  their  position  within 
the  overall  system,  such  as  getting  lost  or  having  screens  lock  up.  As  these 
problems  were  caused  by  a  lack  of  familiarity  with  the  system,  more  guidance 
to  the  user  should  improve  the  situation.  The  capability  to  trap  errors  is  also 
available  in  the  software  package.  This  feature  could  be  used  to  capture 
incorrect  user  responses  and  prevent  lock  up. 

Several  users  suggested  that  a  back-up  capability  be  added  to  the 
program  that  would  allow  the  user  to  go  back  to  the  previous  screen.  Another 
suggestion  was  for  context  sensitive  help,  whereby  the  help  provided  to  the 
user  would  depend  on  their  location  within  the  program.  The  normal 
observation  or  the  default  choice  during  the  fault  isolation  procedure  could  also 
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be  highlighted  to  the  user.  All  of  these  suggestions  should  be  considered  for 
inclusion  in  future  versions  of  the  pwitotype. 

One  user  suggested  that  the  systenx  would  be  more  user  Mendly  witli  e 
pen.  The  authors  and  those  users  who  saw  the  prototype  demonstrated  on  the 
notepad  system  agree  with  this  comment.  Overall,  the  ease  of  use  of  the 
system  scored  highly  and  several  significant  comments  on  additional  features 
were  received  from  the  survey.  Incorporation  of  these  features  and  the  use  of 
the  Notepad  system  should  ensure  that  the  system  becomes  easier  to  use  in 
future  versions  of  the  prototype. 

Suitability  of  User  Interface 

A  total  of  J.l  questions  were  used  to  assess  the  adequacy  of  the  user 
interface.  This  area  accounted  for  the  most  survey  questions  because  the 
success  or  failure  of  the  system  usually  depends  on  whether  or  not  the  user 
likes  using  the  system.  The  following  questions  were  used  to  assess  the  usei" 
interface: 

Q6.  The  screen  layout  is  suitable  for  the  system’s  purposes.  (Mean  =  4.37) 

Q7.  The  overall  textual  presentation  (font  size,  color,  and  typeface)  is  easy  to 
read.  (Mean  =  4.98) 

Q8.  The  use  of  on-screen  buttons  is  acceptable.  (Mean  =  4.99) 

Q9.  The  location  of  on-screen  buttons  is  suitable.  (Mean  =  5.22) 

QIO.  The  subjects  to  which  tl?.e  buttons  are  linked  are  appropriate.  (Mean  = 
4.57) 

Qll.  The  way  in  which  information  is  presented  in  different  windows  is 
suitable.  (Mean  -  4.68) 
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Q12.  The  use  of  different  colors  to  highlight  waitings,  cautions  etc.  is  suitable. 
(Mean  »  5.39) 

Q13.  The  way  that  the  system’s  graphics  are  presented  is  appropriate  to  users 
of  all  experience  levels.  (Mean  =  4.12) 

Q14.  The  method  of  control  (pointer/mouse)  is  suitable.  (Mean  =  6.25) 

Q15.  Tho  provision  of  scroll  bars  permits  easy  viewing  of  text  and  graphic 
information  which  is  larger  than  the  display  window.  (Mean  -  4.93) 

Q16.  The  maximize  capability  which  enlarges  the  graphic  images  to  full  screen 
is  a  useful  feature.  (Mean  =  4.77) 

The  prototype  scored  highly  on  all  11  questions.  Significant  comments  received 
on  the  user  interface  with  regards  to  screen  layout,  buttons,  colors,  pointing 
device,  graphics,  and  the  scroll/maximize  function  are  now  discussed. 

The  split  screen  layout  approach,  where  procedural  information  was 
displayed  on  the  left  hand  side  of  the  screen  and  linked  information 
(text/graphic)  presented  on  the  right  hand  side  of  the  screen,  was  liked  by  most 
users.  One  improvement  suggested  by  a  respondent  was  to  include  a  list  of 
available  graphics  in  a  pull-down  menu  where  it  could  be  accessed  as  required. 
This  recommendation  should  be  considered  in  the  next  version  of  the  prototype 
because  having  a  pull-  down  menu  of  available  graphics  would  also  improve  the 
useability  of  the  system  with  technicians  of  different  experience  levels. 
Another  suggestion  was  that  if  procedural  information  referenced  a  graphic 
then  the  graphic  should  be  displayed  automatically  next  to  the  referencing 
text.  This  feature  should  be  implemented.  However,  where  more  graphics 
exist  than  can  be  displayed  in  the  area  provided,  the  user  should  be  advised 
that  additional  graphical  information  is  available. 
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Most  users  of  the  prototype  were  not  impressed  with  the  quality  of  the 
graphics  displayed.  This  is  a  direct  result  of  the  designers  not  having  access 
to  a  suitable  optical  scanner.  Future  prototypes  should  have  higher  quality 
graphics.  One  respondent  suggested  that  a  graphic  should  not  be  larger  than 
the  screen  size.  The  display  of  graphics  is  an  area  that  requires  further 
research  regarding  the  best  method  for  display  on  a  computer  screen.  For 
example,  what  is  the  best  method  for  displaying  the  information  that  is 
normally  shown  on  several  fold-out  pages  of  a  schematic  diagram  -  a  one 
screen  graphic  enlarging  segments  of  the  drawing  or  a  multi-screen  graphic 
implementing  scroll  bars? 

The  scroll  and  maximize  features  displayed  by  the  prototype  were  not  as 
successful  as  the  designers  hoped.  The  main  problem  was  that  the  right  hand 
side  scroll  bar  disappeared  under  the  frame  when  a  graphic  was  maximized  to 
full  screen,  thus  preventing  this  feature  from  being  properly  utilized.  The 
problem  was  caused  by  a  bug  in  the  software  package  being  used.  However, 
future  releases  of  the  software  should  overcome  this  problem. 

While  the  operation  of  the  buttons  was  accepted  by  most  users,  two 
improvements  were  suggested.  First,  the  buttons  should  be  dynamic  and 
context  sensitive,  whereby  the  information  linked  to  should  be  dependent  on 
the  location  within  the  system.  Second,  the  button  functions  could  be  placed 
in  a  pull-down  menu,  which  would  also  allow  more  button  features  to  be  added 
without  crowding  the  current  layout. 


63 


A  variety  of  comments  were  received  regarding  the  colors  used  within  the 
system,  indicating  that  the  choice  of  colors  is  definitely  a  matter  of  personal 
preference.  Several  factors  need  to  be  considered  in  future  prototypes.  First, 
although  the  current  version  of  the  Notepad  computer  does  not  have  a  color 
display,  a  commercial  color  version  is  expected  within  the  next  few  years. 
Second,  as  the  system  is  designed  for  use  in  various  lighting  conditions  in 
workshop  and  flightline  environments,  a  review  of  existing  research  is  required 
to  optimize  the  screen  presentation  (i.e.,  colors,  fonts,  and  formats). 

As  stated  previously,  most  respondents  believe  tliat  a  pen  would  be  more 
user  fnendly  than  a  mouse  and  that  the  pointing  device  would  be  easier  to  use 
when  the  requirement  to  single  or  double  click  to  select  an  input  choice  is 
removed  from  the  system.  Overall,  the  user  interface  received  high  scores  from 
respondents.  The  incorporation  of  several  suggestions  should  improve  the  user 
interface  further. 

User  Level  Adequacy 

The  following  question  was  used  to  evaluate  whether  the  prototype  was 
suitable  for  use  by  technicians  having  a  range  of  experience  levels  from  novice 
to  expert: 

Q21.  The  system  is  suitable  for  users  with  different  levels  of  knowledge/ 

experience.  (Mean  =  4.49) 

Generally,  survey  respondents  felt  that  the  system  was  suitable  for  use 
by  technicians  with  a  range  of  experience  levels.  However,  this  area  is  one 
where  the  designers  believe  improvements  are  possible.  Future  versions  of  the 
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prototype  could  include  pull-down  menus  which  allow  selection  of  available 
graphics  and  current  button  functions.  The  system  would  then  cater  to 
technicians  with  different  experience  levels  by  giving  them  the  opportunity  u> 
access  as  much  of  the  available  information  as  they  desire. 

The  level  of  information  provided  could  also  be  an  automated  function. 
A  central  file  recording  user  experience  could  be  invoked  when  a  user’s 
identification  was  entered  at  start-up.  Information  regarding  the  user’s  level 
of  experience  could  be  used  to  download  the  most  appropriate  data  and  set  the 
applicable  levels  of  information  display. 

Correctness  of  Converted  Procedure 

The  methodology  for  conversion  of  the  existing  technical  publications  into 

electronic  format  was  assessed  using  the  following  questions: 

Q17.  The  steps  of  the  maintenance  procedure  are  segmented  into  logical 
presentation  groups.  (Mean  =  4.77) 

Q19.  The  hypertext  links  provide  access  to  additional  information  at 
appropriate  stages  in  the  analysis.  (Mean  =  4.50) 

Q22.  The  system  correctly  implements  the  fault  analysis  procedure  as 
presented  in  the  technical  order.  (Answer  Not  Applicable  if  user  does  not 
have  access  to  the  aircraft  publications.)  (Mean  =  4.750,  95%C1  2.75  - 
6.75) 

The  purpose  of  question  17  was  to  test  the  approach  of  "chunking" 
information  into  logical  presentation  groups.  'This  approach  was  successfully 
accepted  by  respondents.  The  only  significant  negative  comment  received  for 
question  17  was  that,  because  the  tasks  were  not  labelled,  the  respondent 
could  not  keep  track  of  where  he  was  in  the  overall  system.  This  loss  of 
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visibility  of  the  current  position  has  been  previously  mentioned.  However,  the 
prototype  could  be  further  improved  by  placing  appropriate  titles  on  tasks, 
graphics,  and  all  referenced  links.  Another  possible  improvement  would  be  to 
use  a  small  area  of  the  screen  to  graphically  display  the  current  status  of 
overlayed  pages.  Tliis  display  would  help  the  user  to  identify  whr  t  hypermedia 
links  were  currently  activated. 

Question  19  was  used  to  assess  whether  the  hj^pertext  information  was 
being  correctly  linked  and  whether  sufficient  hypei’text  links  were  available  to 
the  user.  Generally,  respondents  were  pleased  with  the  hyjiertext  capability 
of  the  model.  Several  comments  that  were  provided  require  further  discussion. 

A  very  good  comment  from  one  of  the  F-15E  SPO  respondents  was  that 
a  specific  maintenance  task  (e.g.,  aircraft  safe  for  maintenance)  required  to 
support  the  checkout  procedure  should  not  merely  discuss  the  performance  of 
the  task  at  an  overview  level  but  should  actually  provide  step-by-step 
instructions  to  perform  the  task.  The  incorrect  linking  in  this  instance  was 
caused  by  the  different  levels  of  information  contained  within  the  various 
sections  of  the  technical  publications  and  the  inexperience  of  the  prototype 
designers  with  the  F-15E  NLG.  The  important  lesson  learned  was  that  the 
correct  level  of  data,  be  it  system,  subsystem,  or  sub-subsystem,  should  be  used 
when  establishing  a  referencing  link.  The  level  of  the  referenced  data  should 
be  determined  by  the  context  of  the  information  at  the  origin  of  the  link. 

Several  other  comments  concerned  the  fact  that  some  of  the  links  led  to 
dead  ends  and  that  there  was  a  need  for  referential  linking.  The  time 
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available  precluded  extensive  development  of  all  the  necessary  links.  The 
prototype’s  main  purpose  was  to  display  a  concept  and  to  provide  a 
demonstration  of  the  linking  possibilities  and  their  operation.  In  the  next 
stage  of  the  prototype,  the  quantity  of  the  links  available  will  likely  be 
expanded.  Full  authoring  would  be  required  to  complete  the  linking  process. 

Question  22  attempted  to  determine  if  the  knowledge  contained  within 
the  knowledge  base  of  the  integrated  expert  system  correctly  reflected  the 
hardcopy  publication.  Because  of  the  inaccessibility  of  F-15E  publications  to 
all  suivey  respondents,  a  very  low  response  rate  (four  out  of  20)  was  received, 
and  the  answer  to  this  question  was  inconclusive.  The  comments  received 
from  the  F-15E  members  who  evaluated  the  prototype  were  that  the  knowledge 
contained  within  the  system  was  correct  but  that  the  system  did  not  contain 
suiflcient  levels  of  detail  for  a  true  evaluation.  The  correctness  of  the 
knowledge  within  the  integrated  expert  system  should  be  tested  in  an  actual 
field  environment  by  a  technician  attempting  to  isolate  existing  faults. 

Overall,  the  procedure  used  for  the  conversion  of  the  hardcopy  technical 
publications  into  electronic  format  appears  to  be  correct.  However,  further 
prototype  development  and  actual  field  testing  is  required. 

Completeness  of  Converted  Pro  edure 

The  following  question  was  used  to  assess  whether  the  entire  knowledge 
necessary  to  perfoim  a  task  was  contained  within  the  knowledge  base  of  the 
integrated  expert  system: 
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Q23.  The  system  pnmdes  access  to  the  knowledge  necessary  for  the  F-  15E 
Nose  IiAnding  Gear  Extraction  and  Retraction  Checkout  Procedure. 
(Answer  Not  Applicable  if  uaer  does  not  have  access  to  the  aircraft 
public^itions.)  (Mean  »  4.67/ 

A  low  response  rate  to  this  question  (six  out  of  29)  was  also  received. 
However,  tlie  result  of  the  statistical  analysis  was  that  the  respondents  agreed 
that  sufficient  knowledge  was  contained  within  the  system  to  represent  the 
complete  knowledge  required  to  perform  the  soeciiic  task  associated  with  the 
landing  gear  esxraction  and  retraction  checkout  procedure. 


Efficiency 

The  following  question  was  used  to  compare  the  prototype  with  the 
hardcopy  technical  publications  to  determine  if  the  prototype  would  be  more 
efficient  than  the  existing  method: 

Q24.  The  system  is  more  efficient  than  using  conventional  technical  order 
publications.  (Answer  Not  Applicable  if  user  does  not  have  access  to  the 
aircraft  publications.)  (Mean  =  4.14,  95%CI  =  3.31  -  4.98) 

Again,  the  low  response  rate  (seven  out  of  20)  to  this  question  resulted 

in  an  inconclusive  answer.  Ideally,  the  system  should  be  field  tested  and  a 

comparison  made  between  the  two  methods  based  on  the  measurement  of  time 

to  make  serviceable.  The  results  are  best  summarized  by  the  following 

comment: 

Difficult  to  assess.  There  are  advantages  over  paper  TOs,  but  the 
difficulties  of  computers  have  not  been  experienced.  [Could]  answer 
better  after  actual  flightline  test. 
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>f  Results 


Reliability  and  consistency  of  the  prototype  were  evaluated  using  the 
following  question: 

Q25.  The  system  provides  expected  results.  (Mean  =  4.62) 

Most  survey  respondents  answered  this  question  favorably.  They 
commented  that  the  system  followed  the  papercopy  fault  isolation  publication 
and  led  the  technician  to  probable  repairs.  However,  this  criterion  requires 
further  assessment  in  the  field  environment. 

Overall  Usefulness  of  the  System 

The  following  seven  questions  were  used  to  assess  the  overall  usefulness 
of  the  protot)rpe: 

Q18.  By  limiting  the  user  to  a  direct  path  through  the  analysis  procedure,  the 
system  eliminates  the  complexities  of  fault  analysis.  (Mean  -  4.32) 

Q20.  The  system  is  likely  to  be  more  acceptable  to  the  technician  (end  user) 
than  the  current  hardcopy  publications.  (Mean  =  3.84,  95%CI  =  3.37  - 
4,30) 

Q26.  Overall,  the  system  is  useful.  (Mean  =  4.83) 

Q27.  The  system  would  be  useful  in  a  workshop  environment.  (Mean  =  4.85) 

Q28.  The  system  would  be  useful  in  a  flightline  environment.  (Mean  =  4.20) 

Q29.  The  system  would  be  useful  to  manage  technical  maintenance.  (Mean  = 
4.82) 

Q30.  The  system  could  be  used  to  manage  technical  publications.  (Mean  = 
4.99) ' 

The  purpose  of  question  18  was  to  assess  the  usefulness  of  the  system  in 
eliminating  the  complexities  of  fault  isolation  and  to  determine  if  a  novice 
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technician  could  perform  the  task.  Even  though  the  net  result  from  the  survey 
was  agreement  that  the  complexities  of  fault  isolation  v/ere  reduced,  two 
noteworthy  conunents  were  received.  One  technician  noted  that  the  user 
should  be  guided  and  not  handcuffed  into  a  certain  fault  path,  and  another 
individual  claimed  that  fixed  fault  trees  are  obsolete. 

The  authors  do  not  agree  that  fixed  fault  trees  are  obsolete.  They  do, 
however,  acknowledge  that  a  small  percentage  (hopefully)  of  faults  will  not  be 
solved  by  using  the  existing  LTTAs.  An  expert  system  is  only  as  good  as  the 
knowledge  stored  within  its  knowledge  base.  The  system  therefore  needs  to 
be  designed  to  cope  with  the  exception.  The  prototype  coped  with  the  exception 
by  requiring  the  user  to  consult  his  supervisor.  An  improvement  would  be  for 
the  system  to  trace  the  path  taken  to  reach  the  exception  situation.  This 
capability  would  allow  for  back-tracking  and  further  (higher  level)  fault 
analysis  by  suitable  experts. 

Question  20  was  used  to  assess  the  preference  of  the  respondents  for 
either  the  prototype  or  the  existing  technical  publications.  The  survey  results 
indicated  a  marginal,  but  not  conclusive,  preference  for  the  prototype. 
Comments  favoring  the  existing  system  were  "that  they  were  better  for  users 
who  weren’t  quite  sure  where  to  look"  and  that  "hardcopy  will  still  be 
required."  An  element  of  resistance  to  change  was  also  noted  in  the  survey 
responses.  Fu  rther  field  testing  of  the  prototype  will  be  required  to  determine 
the  technicians'  preferences.  The  statement  that  "overall,  the  system  is  useful" 
was  highly  scored  by  all  respondents  (range  of  scores  received  were  from  3.5 
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to  6.0),  which  provides  encouragement  for  continued  development  of  the 
prototype. 

Questions  27  and  28  asked  the  respondents  whether  the  system  would 
be  useful  in  workshop  and  flightline  environments.  General  agreement  was 
reached  that  the  system  was  useful  in  a  workshop  environment,  but 
respondents  were  not  as  convinced  of  the  usefulness  of  the  system  in  the 
flightline  environment.  Most  respondents  believed  that  technology  suitable  for 
the  flightline  was  not  yet  available.  They  observed  that,  for  the  system  to  be 
useful  in  an  outdoor  environment,  better  visibility  in  sunlight,  a  long  lasting 
battery,  compact  sixe,  ruggedized  design,  and  portability  were  needed.  The 
NCR  3125  Notepad  computer  has  most  of  these  technological  advancements. 
However,  the  battery  has  only  a  four  hour  capacity  under  normal  use,  and  the 
computer  is  not  ruggedized.  Also,  while  a  ruggedized  version  is  under 
development  by  NCR,  such  a  version  is  already  ofFe/ed  by  the  Slate 
Corporation.  Therefore,  a  different  result  to  question  28  may  have  occurred 
if  the  Notepad  computer  had  been  used  in  the  evaluation  phase. 

Questions  29  and  30  asked  the  respondents  if  the  system  would  be  useful 
to  manage  technical  maintenance  and  technical  publications.  Respondents 
agreed  that  the  system  was  useful  and  has  the  capability  but  needs  some  work. 

IDESM  Evaluation 

Overall,  the  evaluation  has  resulted  in  approval  of  the  stage  1/2  prototype 
and  provided  a  list  of  suggested  improvements.  The  survey  has  demonstrated 
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that  the  concepts  underlying  the  IDESM  are  valid.  Incorporation  of  some 
minor  moditications  to  the  model  and  development  of  a  second  version  of  the 
prototype  should  now  proceed.  Further  development  and  testing  of  the 
model/prototype  is  required  to  determine  if  the  IDESM  can  achieve  the  desired 
goals  of  reducing  the  technical  training  commitment,  providing  more  effective 
fault  isolation  by  technicians  with  reduced  experience  levels,  and  producing 
higher  quality  maintenance. 

Summary 

This  chapter  presented  the  results  obtained  from  the  survey.  A 
statistical  analysis  of  the  results  was  used  to  evaluate  the  prototype  in  the 
eight  performance  areas  identified  in  Chapter  3.  All  significant  comments 
received  from  the  survey  were  addressed  and  improvements  to  the  protot3T)e 
were  identified.  The  major  improvements  to  the  prototype  include:  providing 
more  guidance  to  the  user  (instructions  and  extended  and  context  sensitive 
help  screens),  providing  a  back-up  (to  previous  screen)  feature,  placing  lists  of 
available  graphics  and  buttons  in  pull-down  menus,  enhancing  the  graphics, 
and  providing  context  linking  of  the  hypermedia  information.  The  overall 
assessment  of  the  prototype,  and  hence  the  model,  was  agreement  that  the 
protot3q)e  was  an  improvement  over  the  existing  fault  isolation  manuals. 
Further  development  and  testing  of  the  model/prototype  should  now  proceed. 


V,  ConcluBions  and  Recommendatioiifl 


Introduction 

Tliis  chapter  presents  the  authors’  conclusions  with  respect  to  rhe 
research  conducted  to  meet  the  objectives  stated  in  Chapter  1.  Also,  areas  for 
future  development  of  the  Integrated  Diagnostic  Expert  System  Model 
(IDESM)  and  topics  for  further  research  are  recommended. 

The  purpose  of  this  research  was  to  develop  a  model  to  convert  existing 
hardcopy  maintenance  manual  fault  isolation  procedures  into  electronic  format 
and  to  partially  automate  the  favilt  isolation  process.  A  comprehensive  model 
was  developed  which  comprised  four  major  components:  a  user  interface,  an 
expert  system  submodel,  a  hypermedia  submodel,  and  a  submodel  integration 
shell.  Using  this  mcdel,  the  F-15E  nose  landing  gear  fault  isolation  procedures 
were  transformed  into  an  expert  system.  An  existing  hypermedia  information 
base,  developed  using  Casseirs  SHADM,  was  expanded  and  subsequently 
integrated  with  the  expert  system  to  form  a  prototype  application. 

Engineering  and  maintenance  personnel  at  Wright-Patterson  Air  Force 
Base  evaluated  the  prototype  in  eight  performance  areas;  ease  of  use, 
suitability  of  user  interface,  user  level  adequacy,  correctness  of  converted 
procedure,  completeness  of  converted  procedure,  efficiency,  reliability  and 
consistency  of  results,  and  the  overall  usefulness  of  the  system.  An  analysis 
of  the  survey  responses  demonstrated  favorable  acceptance  of  the  promtype 
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and  validated  the  IDES  model.  Therefore,  the  following  conclusions  can  be 
drawn  from  this  research. 

Conclusions 

1.  The  IDES  model  is  suitable  for  converting  LlTAs  into  rule-based 
expert  systems.  Since  1976  the  msgority  of  USAF  fault  isolation  manuals, 
including  the  F-15E  manuals,  have  been  based  on  Logic  Tree  Troubleshooting 
Aids.  Because  the  F-15E  NLG  fault  isolation  procedures  were  easily  converted 
into  an  integrated  expert  system  using  the  IDES  model,  a  major  benefit  to  the 
Australian  Defence  Force  from  this  research  is  that  any  publication  which  has 
the  fault  isolation  procedures  expressed  in  LTTA  format  can  now  be  converted 
into  an  expert  system  using  the  developed  model. 

2.  The  PMA  concept  is  a  suitable  platform  on  which  to  base  further 
expert  system  development.  Based  on  current  developments  in  information 
systems  technology  and  the  research  being  undertaken  by  Armstrong 
Laboratories,  a  portable  maintenance  aid  was  considered  to  be  the  most  likely 
device  that  the  technician  would  use  in  the  work  environment.  Consequently, 
the  user  interface  specification  of  the  IDESM  was  designed  around  the  PMA 
concept.  Although  the  Notepad  computer  was  unavailable  for  the  survey,  the 
enthusiastic  response  received  when  demonstratiiiig  the  prototype  to  several 
agencies  at  Wright-Patterson  AFB  endorsed  the  PMA  concept.  A  second 
benefit  to  the  ADF  from  this  research  is  that  future  technical  information 
systems  can  now  be  developed  utilizing  the  PMA  concept. 
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3.  Draft  Military  ^pecilKcatioti  MIL-M-GCSFUI  provides 
comprehensive  guidance  for  user  interface  desispn.  The  user  interface 
developed  in  this  research  was  consistent  with  (draft)  MIL-M~GCSFUI  (9)  and 
will  prove  useful  in  the  development  of  other  similar  types  of  expert  systems. 
Extensive  design  effort  on  the  user  interface  was  undertaken  because  of  its 
importance  to  end  product  acceptance  by  the  user.  In  order  to  simplify  the 
task  and  minimize  user  interaction,  input  was  limited  to  "point  and  click" 
functions  rather  than  keyboard  entry.  The  split  screen  layout  presented  text 
and  graphics  in  a  format  similar  to  its  hardcopy  predecessor.  Although  some 
training  will  be  required  to  operate  the  system,  technicians  who  currently  use 
the  existing  publications,  should  not  require  training  to  become  familiar  with 
this  presentation  format.  Presentation  of  the  information  was  enhanced  by  the 
use  of  colors  (where  available),  and  features  such  as  scrolling  and 
maximization  were  included.  The  survey  results  demonstrated  that  the  user 
interface  was  highly  accepted  by  the  user  population.  The  third  benefit  from 
this  project  is  that  a  useable  product,  which  may  be  subsequently  refined  by 
continuing  research  into  human-computer  interaction,  has  been  developed. 

4.  The  modular  design  of  the  current  IDESM  prototype  allows  it  to 
be  easily  expanded  to  a  full-scale  standardized  system.  Considerable 
research  into  the  existence  of  models  that  convert  fault  isolation  procedures 
into  expert  systems  was  not  productive.  Therefore,  Seim’s  principles  of 
software  design  were  used  in  the  construction  of  the  expert  system  and 
hypermedia  submodels.  Modularity  was  introduced  by  structuring  the  expei  t 
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system  into  several  layers  of  knowledge  bases.  The  layers  corresponded  to  the 
system,  subsystem,  and  sub-subsystem  technical  assignment,  as  detailed  by 
MILrSTD-18G8.  Each  layer  comprised  task  specific  modules,  each  of  which 
contained  the  knowledge  required  to  perform  that  particular  fault  isolation 
procedure,  llie  h^rpermedia  submodel  was  based  primarily  on  graphical  and 
textual  information  contained  within  the  existing  hardcopy  manuals.  However, 
the  potential  to  include  other  forms  of  media,  e.g.,  video  and  audio,  was 
recognized.  Because  of  the  necessity  to  reference  common  information  (e.g., 
drawings,  warnings,  and  cautions)  throughout  the  text,  hardcopy  manuals 
contain  redundant  information.  The  hypermedia  submodel,  which  stores 
information  in  modules,  overcomes  this  redimdancy  problem  by  allowing  the 
shared  use  of  the  modular  information.  The  design  principles  of  modularity 
and  shared  use  of  modules  provide  significant  benefits  in  minimizing  the 
memory  storage  requirements  of  information  systems  and  simplifying  software 
maintenance.  Further,  software  progranuning  templates  could  be  developed 
for  each  of  the  levels/functions  in  the  IDESM,  providing  faster  development 
and  greater  standardization  of  integrated  expert  systems. 

5.  KnowledgePro®  for  Windows  is  a  suitable  off-the-shelf  software 
package  for  continued  development  of  the  integration  shell.  The  final 
component  of  the  IDESM  was  the  integration  shell  submodel,  which  allows  the 
user  to  transparently  access  information  within  the  hypermedia  and  expert 
system  knowledge  bases.  KPW  was  selected  for  this  application  because  of  its 
ability  to  perform  hypertext  and  expert  system  functions  and  its  versatile 
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display  capabilities.  In  addition,  it  was  easy  to  leani  and  use.  As  KPW  is  a 
relatively  new  product,  the  software  developers  at  Knowledge  Garden  are  keen 
to  improve  the  performance  of  their  product.  Significant  improvements  have 
been  made  to  the  product  in  the  12  months  since  FLTLT  Cassell  completed  nis 
research.  During  the  development  of  this  prototype,  the  authors  provided  some 
recommendations  for  product  improvement.  Consequently,  the  company  is 
interested  in  providing  the  next  release  of  software  for  alpha  site  testing  with 
the  protot:/pe.  The  next  version  of  KPW  will  permit  the  compilation  of 
developed  programs  into  C++  executable  code  for  distribution.  This  is  expected 
to  improve  the  execution  time  of  the  application  software  by  a  factor  of  10  to 
30  times.  The  benefits  from  co-operative  development  of  a  hypermedia/expert 
system  softv/are  product  could  be  significant  to  the  ADF.  Possible  benefits 
include  cognizant  development  of  a  product  which  considers  military  unique 
requirements  such  as  MIL-M-GCSFUI,  CALS,  and  the  PMA  concept. 
Consequently,  this  proposal  will  be  forw'arded  for  consideration  by  the  relevant 
authorities. 

Recommendations  for  Further  Research 

Recommendations  from  this  research  are  made  in  two  areas  -  those  that 
relate  directly  to  the  prototype  and  those  that  concern  expert  system 
development  in  general.  First,  the  prototype  should  undergo  further 
development  and  testing.  As  a  result  of  ongoing  research  and  comments 
received  from  survey  respondents,  several  improvements  to  the  prototype  were 
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suggested  in  Chapter  4.  These  improvements  include  the  use  of  single  click  in 
place  of  double  click,  improvements  to  the  maximization  function,  the  inclusion 
of  a  pull-down  menu  bar,  graphics  enhancement,  the  addition  of  a  back-up  (to 
previous  step)  feature,  an  error  trapping  capability,  and  a  context  sensitive 
help  feature.  A  field  test  using  actual  faults  is  recommended  to  more  fully 
evaluate  the  correctness  and  completeness  of  the  information  within  the 
prototype.  Some  of  the  criteria  that  need  further  evaluation  include: 

a.  success  rate  and  accuracy  of  isolating  the  fault, 

b.  effectiveness  of  reasoning  displayed  by  system,  and 

c.  improving  the  technician’s  efficiency  (evidenced  by  any  noticeable 

reduction  in  time  to  make  serviceable). 

Second,  the  coupling  of  the  IDESM  with  existing  ADF  management 
information  systems,  e.g.  maintenance  management  and  supply  systems,  to 
provide  the  next  level  of  system  integration  should  be  investigated.  The  process 
of  recording  th  e  maintenance  action  (repair/replacement  of  components)  in  the 
RAAF’s  Computer  Aided  Maintenance  Management  system  and  demanding  the 
replacement  components  through  the  Defence  Supply  Retail  Management 
System  are  two  possibilities.  These  processes  are  the  next  logical  steps  in 
providing  the  technician  with  access  to  integrated  information  systems  and 
providing  a  paperfree  technical  environment. 

Third,  the  authors  recommend  that  the  expert  system,  hypermedia,  and 
integration  submodels  undergo  further  development.  As  part  of  this  ongoing 
development,  research  into  the  storage  and  processing  of  photographic  quality 
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graphics  and  audioArideo  iriformation  is  recommended.  Also  as  the  information 
available  within  the  integrated  system  becomes  larger,  the  performance  of  the 
system  concerning  peak  loads,  access  times,  and  correct  linkage  of  information 
needs  evaliiation  and  possible  refinement. 

Fourth,  as  stated  in  Chapter  4,  a  candidate  for  further  research  is  the 
graphical  presentation  of  information  on  small  screen  devices.  Research  is 
required  to  determine  the  optimal  methods  for  displaying  schematic  and  wiring 
diagrams.  These  graphics  are  normally  larger  in  size  than  one  screen,  too 
complex  to  read  when  reduced  to  one  screen,  and  too  difficult  to  comprehend 
when  segmented.  Because  the  trend  to  miniaturize  computing  devices  will 
continue,  more  emphasis  on  the  effectiveness  of  graphical  display  methods  will 
be  required. 

Finally,  the  authors’  inexperience  >vith  the  software  package  (KPW) 
prevented  them  from  fullj'^  utilizing  the  object-oriented  programming  (OOP) 
capabilities  of  the  software.  Therefore,  OOP  should  be  investigated  to 
determine  its  applicability  for  structuring  knowledge  in  an  expert  systeiri 
environment. 

Summary 

Several  significant  findings  have  resulted  from  this  research.  First,  the 
IDESM  has  been  developed  which  enables  fault  isolation  procedures  recorded 
in  the  LTTA  format  to  be  converted  into  expert  systems.  Second,  the  model 
provides  for  integration  of  the  expert  systems  with  a  hypermedia  base  and  is 
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suitable  for  integpration  with  other  information  systems.  The  success  of  the 
developed  prototsnpe,  and  hence  the  model,  indicates  that  this  research  will  be 
beneficial  to  the  ADF  and  other  military  organisations.  Finally,  the  following 
suggested  improvements  to  the  prototype  have  already  been  incorporated:  pull¬ 
down  menu  capability,  more  user  guidance  in  the  form  of  window  titles,  and 


correct  linking  of  information. 


Appendix  A:  Sample  File  Formats 


Faultcode  Module  Template 

*  ♦  *  Sets  screen  title  ♦  *  * 

8et_title(?wText,’<Procedure  Title>’). 

*  ♦  ♦  Loads,  runs,  and  unloads  warning  module/screens  *  *  * 
load  C<waming  screen  filename>’,topicl). 

topicK). 

reniove_topic  (topicl). 

*  *  *  Displays  introductory  text  *  *  * 
text  ('#e 

Test  indicator  2  not  turning  white  is  caused  by 
one  of  the  below: 

a.  NLG  down  limit  switch; 

b.  NLG/door  selector  valve; 

c.  NLG  diode  panel; 

d.  aircraft  wiring.’). 

*  *  *  Open  window  -  ask  question  -  close  window  *  *  * 
window  (,1, 16,47 ,6,’NLG  forward  door’,[child, visible, thinframe, 

titlebar, 8ho>vChildren,8iblings],?wText,blue,white„). 

ask  (’  Did  NLG  forward  door  open?’,  re8ult_nlg,  [YES,NO]). 
ciose_window(). 

*  *  *  set  up  bvanching  according  to  LTTA  *  *  * 
if  ?result_nlg  =  YES 

then  do  (8tep_nlg_ye8) 
else  do  (8tep_nlg_no), 

*  *  *  The  YES  branch  ♦  *  * 

Topic  ’Btep_nlg_ye8’. 

*  ♦  Displays  introductory  text  *  *  * 
text  (’#e 

1.  Hydraulic  off. 

2.  Install  NLG  forward  door  groimd  safety  pin 
#m(05-10-12)#m. 
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3.  Manually  press  NLG  down  limit  switch.’). 

*  *  *  Open  window  -  ask  question  -  close  window  *  *  * 
window  (, 1,15,47, 6, ’Indicator2’, [child, visible, thinframe, 

titlebar  ,8howChildren,siblings],?wText,blue,wlnte„). 

ask  (’  Does  test  indicator  2  on  secondary  #n  power  system  test  set  turn  white?’, 

re8ult2,  [YES,NO]). 

clo8e_window(). 

next  branch  according  to  LTFA  if  applicable  ♦  *  * 

if . 

then . 

else . 

*  *  *  close  this  module  &  return  to  calling  module  *  *  * 
and  exit  (next_kb). 

end.  (*  step  nig  yes  *) 

*  ♦  ♦  The  NO  branch  or  other  branches  as  applicable  *  *  * 

Topic  ’8tep_nlg_no’. 

text . 

end.  (*  8tep_nlg_no  *) 

*  *  *  Hypermedia  calling  module  *  *  * 
topic  8uper_mark  (item). 
set_file_po8  (’genmaint.ndx'  ,0). 

:IndexInfo  is  read  (’genmaint.ndx’ ,  concat  (’//’, ?item), ’//’). 

8et_file_po8  (’genmaint.hyp’,  fir8t  (?IndexInfo)). 

:t  is  read_char  (’genmaint.hyp’,  element  (?IndexInfo,  2)). 
markWindow  is  window(,48,l,44,26„[child,vi8ible,thinFrame,vertScroll, 
showChildren,  siblings], ?!wHandle, blue, white,,), 
text  (?t). 
waitO. 

Close_window  (). 
end.  (*  8uper_mark  ’'0 
end. 
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Index  File  Format  Sample 


//Description 

5683  .....  (indicates  start  position  in  61e) 

1160 . (indicates  length  of  information) 

//Operation 

6858 

2179 

//TCTO 

9047 

876 

//Safety 

9934 

821 


Text  File  Format  Sample 

//Description  (This  ASCII  information  starts  at  position  5683  of  the  file) 

The  NLG  is  mounted  in  the  forward  fuselage  at  FS  372.00.  The  gear  is 
normally  extended  by  using  the  wheel-shaped  handle  on  the  LDG  GR  Control 
Panel.  Moring  the  handle  to  DOWN  releases  the  uplatch  and  opens  the  gear 
doors,  which  allows  [etc.] 

//Operation  (This  ASCII  information  starts  at  position  6858  of  the  file) 

The  NLG  and  NLG  forward  doors  are  operated  by  the  LDG  GR  control 
handle.  The  door  is  electrically  sequenced  and  hydraulically  operated.  When 
required  for  ground  maintenance,  the  forward  door  may  be  opened  and  closed 
manually  or  [etc.] 

//TCTO  (This  ASCII  information  starts  at  position  9047  of  the  file) 

The  record  of  applicable  time  compliance  technical  orders  is  a  list  of  all 
TCTOs  which  affect  the  technical  content  (text  or  illustration)  of  this  manual. 
Only  currently  effective  TCTOs  are  listed.  A  TCTO  is  deleted  from  the  list 
when  any  [etc.] 
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Appendix  B:  Model  Evaluation  Questionnaire 


After  using  the  F-15E  FAULT  ANALYSIS  system  please  indicate  the  extent  to 
which  you  agree/disagree  with  the  following  statements.  If  you  score  a 
statement  below  3,  please  provide  supporting  comments  for  your  score.  If  there 
is  insufficient  space,  please  continue  on  the  reverse  side  of  the  page.  Please 
complete  this  questionnaire  and  have  it  ready  for  collection  by  Friday  29  May 
1992.  Thank  you  in  advance  for  your  participation. 

USER  INTERFACE 

1.  The  program  is  easy  to  start. 


STRONGLY  STRONGLY 

DISAGREE  AGREE 


I - 1 - 

1  2 

1 - 

3 

1 

4 

1 

5 

1 

6 

Comments: 

2.  The  program  is  easy  to  use. 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1 

1  2 

Comments: 

1 

3 

1 

4 

1 

6 

1 

6 

3.  Navigating  throiighout  the  fault  analysis  system  is  easy. 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1 - 1  1 - 1 - 1 — 

1  2  3  4  5 

1 

6 

Comments: 
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4.  Sufficient  explanations  (on-screen/documentation)  are  provided  to  use  the 

system. 

STRONGLY 
DISAGREE 

I - T - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


STRONGLY 

AGREE 


Comments; 


5.  Exiting  the  system  is  easy. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 

Comments: 


DISPLAY  STANDARDS 


6.  The  screen  layout  is  suitable  for  the  system’s  purposes. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - -i - 1 

1  2  3  4  5  6 


Comments: 


7.  The  overall  textual  presentation  (font  size,  color,  and  typeface)  is  easy  to 
read. 


STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 
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8.  The  use  of  on-screen  buttons  is  acceptable. 


STRONGLY 

DISAGREE 

STRONGLY 

AGREE 

1  1  1  1 

12  3  4 

1 

5 

1 

6 

Comments: 

9.  The  location  of  on-screen  buttons  is  suitable. 

STRONGLY 

DISAGREE 

STRONGLY 

AGREE 

1  1  1  1 

12  3  4 

1 

5 

1 

6 

Comments: 

10.  The  subjects  to  which  the  buttons  are  linked  are  appropriate. 

STRONGLY 

DISAGREE 

STRONGLY 

AGREE 

1  1  1  1 

12  3  4 

1 

5 

1 

6 

Comments: 

11.  The  way  in  which  information  is  presented  in 
suitable. 

different  windows  is 

STRONGLY 

DISAGREE 

1 - 1 - 1 - 1 - 

— 1 — 

STRONGLY 

AGREE 

1 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 
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12.  The  use  of  different  colors  to  highlight  warnings,  cautions  etc.  is  suitable. 


STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - r- - 1 - 1 - 1 - 1 

1  2  3  4  5  6 

Comments: 


13.  The  way  that  the  system’s  graphics  are  presented  is  appropriate  to  users 
of  all  experience  levels. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments; 


14.  The  method  of  control  (pointer/mouse)  is  suitable. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - \ - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 


15.  The  provision  of  scroll  bars  permits  easy  viewing  of  text  and  graphic 
information  which  is  larger  than  the  display  window. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

[ - \ - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 
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16.  The  maximize  capability  which  enlarges  the  graphic  images  to  full  screen 
is  a  useful  feature. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  I  I  I  I  I 

1  2  3  4  5  6 


Comments: 


CONTENT 

17.  The  steps  of  the  maintenance  procedure  are  segmented  into  logical 
presentation  groups. 


STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1 

1  2 

1 

3 

1 

4 

1 

5 

1 

6 

Comments; 


18.  By  limiting  the  user  to  a  direct  path  through  the  analysis  procedure,  the 
system  eliminates  the  complexities  of  fault  analysis. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - ^ ^ - 1 

1  2  3  4  5  6 

Comments; 


19.  The  hypertext  links  provide  access  to  additional  information  at 
appropriate  stages  in  the  analysis. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 
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GENERAL 


20.  The  system  is  likely  to  be  more  acceptable  to  the  technician  (end  user) 
than  the  current  hardcopy  publications. 


STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1 "  1 

1  2 

1 

3 

1 

4 

1  ' 

5 

1 

6 

Comments: 

21. 

The  system  is  suitable  for  users 
experience. 

with  different  levels  of  knowledge/ 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1  1 

12  3 

1  1  1 

4  5  6 

Comments: 

22. 

The  system  correctly  implements  the  faiJt  analysis  procedure  as  presented 
in  the  technical  order.  (Answer  Not  Applicable  if  user  does  not  have  access 

to  the  aircraft  publications.) 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

12  3 

1"  1  . 1 

4  5  6 

Comments: 
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23.  The  system  provides  access  to  the  knowledge  necessary  for  the  P-  15E  Nose 
Landing  Gear  Extraction  and  Retraction  Checkout  Procedure.  (Answer 
Not  Applicable  if  user  does  not  have  access  to  the  aircraft  publications.) 


STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1  1 

12  3 

1 

4 

1 

5 

1 

6 

Comments: 

24.  The  system  is  more  efficient  than  using  conventional  technical  order 

publications.  (Answer  Not  Applicable  if  user  does  not  have  access  to  the 

aircraft  publications.) 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1  1 

12  3 

1 

4 

1 

5 

1 

6 

Comments: 

25.  The  system  provides  expected  results. 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1  1 

12  3 

i 

4 

1 

5 

1 

6 

Comments; 

26.  Overall,  the  system  is  useful. 

STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1  1 

12  3 

1 

4 

1 

5 

1 

6 

Comments; 
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27.  The  system  would  be  useful  in  a  workshop  environment. 


STRONGLY 

STRONGLY 

DISAGREE 

AGREE 

1  1 

1  2 

1 

3 

1 

4 

1 

5 

1 

6 

Comments: 


28.  The  system  would  be  useful  in  a  flightline  environment. 

STRONGLY 
AGREE 

I - 1 - r - \ - ^ - 1 

1  2  3  4  5  6 


STRONGLY 

DISAGREE 


Comments; 


29.  The  system  would  be  useful  to  manage  technical  maintenance. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  \  I  I  I  I 

1  2  3  4  5  6 


Comments; 


30.  The  system  could  be  used  to  manage  technical  publications. 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I - 1 - 1 - 1 - 1 - 1 

1  2  3  4  5  6 


Comments: 
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Please  feel  free  to  address  any  specific  strengths  or  weaknesses  you 
observe  in  the  package  but  were  not  addressed  specifically  by  the  questionnaire. 
Use  the  space  beiow  or  attach  additional  sheets  as  required. 

Comments: 


Name  (Optional): _ 

Rank  or  Title  : _ 

AFSC  (or  equivalent): _ 

Please  list  any  previous  relevant  experience  (Technical  orders,  F-15E 
maintenance,  hypertext  systems,  expert  systems,  etc.): 
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Appendix  C:  Questionnaire  Responses 


USER  INTERFACE 

Question  1.  The  program  is  easy  to  start. 

Mean  =  5.38,  SD  =  0.84,  95%  Cl  =  4.98  -  5.77,  n  =  20. 

Comments: 

a.  Starting  the  application  from  windows  is  very  easy. 

b.  Once  the  application  is  running,  getting  into  the  contents  of  the  system  is 
also  easy. 

c.  There  should  be  some  sort  of  message  line  to  ’steer’  the  user  through  the 
screens,  e.g.  ’Press  continue  for  next  step.’ 

d.  Automatic  installation  program  woxild  be  an  advantage. 

e.  Screen  opens,  then  appears  to  close  and  reopen. 

f.  No  problem  here. 

Question  2.  The  program  is  easy  to  use. 

Mean  =  4.49,  SD  =  0.98,  95%  Cl  =  4.03  -  4.95,  n  =  20. 

Comments: 

a.  Could  not  back  up  a  screen. 

b.  Had  two  versions  running  at  the  same  time. 

c.  Navigating  through  the  program  is  relatively  easy,  but  I  often  got  lost  in 
the  big  picture,  i.e.  where  1  was  in  the  procedure,  where  I’m  headed.  It’s 
not  real  bad  but  could  use  some  thought. 
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d.  If  you  followed  the  predicted  path  you  were  fine,  but  it  was  difficult  to 
backup  or  find  your  place  if  you  jumped  to  button  options  (several  in  a  row 
without  returning  to  task) 

e.  Would  be  more  user  fnendly  with  a  pen. 

f.  I  kept  having  a  problem  with  the  screens  locking  up.  1  don’t  know  if  this 
was  a  program  malfunction  or  something  1  did.  1  would  have  to  shut  down 
and  restart  to  continue. 

Question  3.  Navigating  throughout  the  fault  analysis  system  is  easy. 

Mean  =  4.18,  SD  =  1.04,  95%  Cl  =  3.69  -  4.66,  n  =  20. 

Comments; 

a.  No  direct  link  between  text  and  graphic.  Step  J  of  procedure  and  graphics 
selection  ’3  level  experience’. 

b.  Needs  some  instructions  to  go  along  with  system.  Just  getting  in  and 
roving  around  doesn’t  seem  adequate  enough. 

c.  Its  OK.  Is  the  normal  observation  defaulted  on  the  fault  analysis  prompts? 
(maybe  they  should  be  to  let  the  user  know  what  the  normal  observation 
is.  A  yes  isn’t  always  a  pass) 

d.  No  way  to  back  up  or  change  a  test  result  (i.e.,  I  selected  no  when  I  should 
have  selected  yes). 

e.  Could  use  titles  for  procedures  on  every  screen  so  you  know  what 
particular  task/test  you  are  performing  and  where  you  are  in  an  overall 
structure. 
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f.  Like  to  explain  and  ask  question  to  show  someone  in  person  while 
operating  the  system. 

Question  4.  Sufficient  explanations  (on-screen/documentation)  are  provided 
to  use  the  system. 

Mean  =  4.08,  SD  =  1.32,  95%  Cl  =  3.46  -  4.70,  n  =  20. 

Comments: 

a.  Had  a  real  hard  time  understanding  graphic  mode/text  mode  difference. 

b.  Decent  for  the  amount  of  content  in  the  system. 

c.  Might  want  to  explore  context  sensitive  help,  i.e.,  rationalization  for 
maintenance  actions.  Again,  might  want  to  provide  a  message  bar. 

d.  How  to  navigate  correctly  was  unclear  if  you  branched  away  to  see  button 
alternatives. 

e.  Where  use  of  ’Continue’,  single/double  click  should  be  given  on  each 
occasion  they  are  required. 

f.  Numerous  times,  I  got  a  spot  where  it  didn’t  do  anything  or  tell  me  to  do 
anything. 

i 

g.  It  may  be  helpful  to  have  index  labelled  or  have  it  automatically  come  up 
when  you  start  the  program. 

Question  5.  Exiting  the  system  is  easy. 

Mean  =  4.82,  SD  =  1.16,  95%  Cl  =  4.27  -  5.36,  n  =  20. 

Conunents: 
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a.  It  is  very  easy  to  exit  with  the  quit  button,  but  you  should  be  able  to  start 
from  introduction  screen  instead  of  checkout  procedure,  if  you  don’t  want 
to  exit  completely. 

b.  Yes,  however  exiting  the  current  screen  for  the  initial  page  should  always 
be  available,  i.e.,  the  first  screen. 

c.  Possible  use  of  an  automatic  exit  icon  may  enhance  this. 

d.  Machine  locked  up  twice  on  the  exit.  Had  to  reboot  to  exit. 

e.  Change  buttons  to  standard  window  type  radio  buttons. 

f.  No  problem  here. 

DISPLAY  STANDARDS 

Question  6.  The  screen  layout  is  suitable  for  the  system’s  purposes. 

Mean  *  4.37,  SD  =  1.17,  95%  Cl  =  3.82  -  4.91,  n  =  20. 

Comments: 

a.  Left  right  format  OK. 

b.  Good.  Same  as  TOs, 

c.  Too  much  screen  real  estate  is  taken  up  with  the  available  graphics  panel 
on  the  right  hand  side,  put  them  in  a  menu. 

d.  Text/graphic  panels  side  by  side  seemed  to  work  well. 

e.  Large  diagrams  are  difficult  to  comprehend  using  a  small  scrollable 
window. 

f.  Maximize  is  useful  but  not  ideal. 
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g.  Screen  needs  to  split  and/or  have  full  screen  capability  with  a  zoom 
addition  for  ease  of  utility. 

Question  7.  The  overall  textual  presentation  (font  size,  color,  and  typeface) 
is  easy  to  read. 

Mean  =  4.98,  SD  =  0.88,  95%  Cl  =  4.56  -  5.39,  n  =  20. 

Comments: 

a.  Color,  should  be  consistent  between  the  leftside  and  the  rightside  of  the 
procedure. 

b.  Lime  green  not  good. 

c.  The  blue  text  color  would  be  hard  to  read  after  a  period  of  time.  Black  may 
be  boring,  but  it  is  still  the  easiest  text  color  to  read. 

d.  Good  color,  font.  Great. 

f.  Warning  color,  for  example,  stands  out  as  important. 

g.  Easy  to  read  in  a  controlled  environment.  May  need  some  added 
capability  (Screen  needs  to  split  and/or  have  full  screen  capability  with  a 
zoom  addition  for  ease  of  utility.) 

h.  Outside  on  a  sunny  day,  it  may  not  be  too  easy  to  read. 

Question  8.  The  use  of  on-screen  buttons  is  acceptable. 

Mean  =  4.99,  SD  =  0.82,  95%  Cl  =  4.61  -  5.37,  n  =  20. 

Comments: 

a.  No  backup  button. 

b.  Yes,  buttons  are  good  but  lets  say  that  on  a  full  graphic  screen  you  have 
buttons  that  may  not  apply  to  that  graphic  in  any  way.  Can  the  buttons 
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change  d3mamically  depending  on  screen  content?  Put  other  buttons  on 
a  pull-down  menu  or  something. 

c.  Navigation  to  them,  back  from  them,  and  between  them  must  be  clear. 

d.  Buttons  should  be  uniform.  Bottom  buttons  only  require  1  click,  others 
reqiure  2  clicks.  Suggest  all  2  click. 

Question  9.  The  location  of  on-screen  buttons  is  suitable. 

Mean  =  5.22,  SD  =  0.69,  95%  Cl  =  4.89  -  5.54,  n  =  20. 

Comments: 

a.  In  good  view  of  user. 

b.  Use  of  a  ’pen’  will  enhance  this  capability. 

Question  10.  The  subjects  to  which  the  buttons  are  linked  are  appropriate. 

Mean  =  4.57,  SD  =  0.92,  95%  Cl  =  4.14  -  5.00,  n  =  20. 

Comments: 

a.  Links  to  description  OK.  General  maintenance  provides  discussion  of 
example  ’Solo  flight,’  but  should  provide  actual  procedures  to  do  the  task. 
When  selecting  (05-00-05)  actual  task  should  appear,  not  just  verse. 

b.  With  training  to  explain  the  function  of  each  button,  the  short  subject 
titles  will  be  acceptable. 

c.  Yes,  buttons  are  good,  but  let’s  say  that  on  a  full  graphic  screen  you  have 
buttons  that  may  not  apply  to  that  graphic  in  any  way.  Can  the  buttons 
change  dynamically  depending  on  screen  content?  Put  other  buttons  on 
a  pull-down  menu  or  something. 
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d.  There  are  a  lot  of  other  good  subjects/links  that  could  be  used  in  addition, 
but  I  realize  this  is  just  a  limited  demonstration. 

e.  For  the  NLG  system,  small  examples  were  linked  to  each  button.  Would 
more  information  for  each  button  be  accessible? 

f.  Use  of  a  ’pen’  will  enhance  this  capability. 

Question  11.  The  way  in  which  information  is  presented  in  different 
windows  is  suitable. 

Mean  =  4.68,  SD  =  0.92,  95%  Cl  =  4.25  -  5.10,  n  =  20. 

Comments: 

a.  Easy  to  move  back  and  forth. 

b.  Available  graphics  window  is  unnecessary  in  my  opinion.  If  a  step  has  a 
graphic  associated  with  it,  show  it;  if  not,  don’t  worry  about  it. 

c.  Might  want  to  think  about  making  all  NotesAVamings/Cautions  pop-up 
windows.  Sometimes  they  appear  this  way,  other  time  they  are  displayed 
on  left  side  of  screen  -  inconsistent. 

d.  Zoom  capability  in  both  text  and  graphics  would  add  more  utility  to  the 
windows. 

e.  It  may  be  better  to  have  safety  items  come  up  automatically  rather  that 
having  the  user  decide  if  he  will  read  them  or  not. 

Question  12.  The  use  of  different  colors  to  highlight  warnings,  cautions  etc. 
is  suitable. 

Mean  =  5.39,  SD  =  0.58,  95%  Cl  =  5.12  -  5.65,  n  =  20. 

Comments: 
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a.  The  way  Cautions  and  Warnings  dominate  the  screen  prior  to  reading  the 
procedures  was  effective. 

b.  Yes.  Red  catches  viewer’s  vision. 

c.  Don’t  know  if  color  makes  a  big  difference,  but  in  this  system  it  works  OK. 

d.  Excellent. 

Question  13.  The  way  that  the  system’s  graphics  are  presented  is 
appropriate  to  users  of  all  experience  levels. 

Mean  =  4.12,  SD  =  1.27,  95%  Cl  =  3.52  -  4.71,  n  =  20. 

Comments: 

a.  It  is  impossible  to  see  an  entire  graphic  on  the  screen  at  one  time. 

b.  In  addition  to  manual  diagrams,  actual  scanned  photographs  would  be 
more  appropriate  in  the  fault  analysis. 

c.  Definitely  need  to  work  in  this  area. 

d.  These  graphics  are  not  acceptable  due  to  quality  of  transferring  from  paper 
to  digital. 

e.  IPB  seems  very  grainy.  Enhancement  to  graphics  might  make  picture 
better, 

f.  An  initial  training  course  would  be  a  must. 

g.  Some  graphics  items  are  a  little  indistinct. 

Question  14.  The  method  of  control  (pointer/mouse)  is  suitable. 

Mean  =  5.25,  SD  =  0.86,  95%  Cl  =  4.84  -  5.67,  n  =  19. 

Comments: 
a.  Mouse  is  OK. 
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b.  A  number  of  selections  with  the  mouse  require  correction  as  many  require 
multiple  selections  to  operate. 

c.  Would  help  later  to  go  to  portable  system. 

d.  I  am  unfamiliar  with  a  program  that  requires  double  clicks  occasionally. 
Further  instruction  would  make  it  easier. 

e.  A  pointer  would  be  a  lot  better. 

Question  15.  The  provision  of  scroll  bars  permits  easy  viewing  of  text  and 
graphic  information  which  is  larger  than  the  display  window. 

Mean  =  4.93,  SD  =  1.15,  96%  Cl  =  4.39  -  5.46,  n  =  20. 

Comments: 

a.  All  graphic  not  able  to  be  on  screen. 

b.  While  the  method  implemented  is  ideal,  the  loss  of  the  right  hand  scroll 
bar  when  window  size  changed  could  present  a  problem.  The  addition  of 
an  expand  to  full  screen  box  for  each  window  would  help  if  scroll  bar 
cannot  be  implemented  on  all  window  sizes. 

c.  Graphic  information  is  difficult  to  comprehend  (i.e.,  ’the  big  picture’)  using 
scroll  bars. 

d.  Graphic  information  should  be  no  larger  than  a  full  screen. 

Question  16.  The  maximize  capability  which  enlarges  the  graphic  images  to 

full  screen  is  a  useful  feature. 

Mean  =  4.78,  SD  =  1.27,  95%  Cl  =  4.15  -  5.42,  n  =  18. 

Comments: 
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a.  Maximize  buttons  hidden  on  right  side  of  screen,  maybe  programming 
problem. 

b.  All  graphic  not  on  screen. 

c.  Never  found  it.  Was  it  clear  where  it  was  and  how  to  use  it. 

d.  But  cannot  always  be  seen  to  select  if  window  not  full  size. 

e.  If  the  quality  of  graphics  and  zoom  capability  were  added. 

f.  Did  not  use  this  option. 

g.  A  zoom  feature  of  the  graphic  would  be  a  welcome  addition.  Also  better 
scans. 

h.  Needs  explanation  of  how  to  do  it.  Someone  who  is  not  computer  literate 
could  have  problems  with  it. 

i.  Did  not  see. 

CONTENT 

Question  17.  The  steps  of  the  maintenance  procedure  are  segmented  into 
logical  presentation  groups. 

Mean  =  4.77,  SD=0.75,  95%  Cl  =  4.41  -  5.14,  n  =  19. 

Comments: 

a.  But  they  were  not  labelled,  so  I  couldn’t  track  where  I  was  and  what  I  was 
accomplishing.  I  felt  trapped  in  a  path. 

Question  18.  By  limiting  the  user  to  a  direct  path  through  the  analysis 
procedure,  the  system  eliminates  the  complexities  of  fault 
analysis. 
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Mean  »  4.32,  SD  *  1.14,  95%  Cl  «  3.78  -  4.86,  n  =  20. 

Gomnients; 

a.  There  wasn’t  enough  experience  in  this  area  to  adequately  rate  fault 
analysis.  When  Yes  was  selected  on  the  first  step,  the  system  locked  up. 

b.  The  user  should  be  guided  but  not  handcuffed  into  doing  some  other 
procedure  that  they  think  may  help  them. 

c.  So  much  more  can  be  accomplished  now  with  expert  systems  and 
interfacing  with  the  aircraft  that  fixed  fault  trees  are  obsolete. 

d.  But  they  were  not  labelled,  so  I  couldn’t  track  where  I  was  and  what  I  was 
accomplishing.  I  felt  trapped  in  a  path. 

e.  Further  insight  to  the  decision  process  at  every  step. 

f.  This  capability  doesn’t  eliminate  the  complexities.  It  will  probably  slow 
the  process  down  rather  than  expedite  matters. 

g.  Would  help  to  back  track  and  know  what  decisions  you  made  to  get  to  the 
point  you  are  at  that  moment. 

h.  Will  this  create  difficulties  in  those  rare  ’exception’  trouble  shooting 
problems?  Will  there  be  enough  detail? 

i.  I  would  like  to  have  seen  trouble  analysis  of  a  system  closer  to  what  I  work 
on  (avionics). 

Question  19.  The  hypertext  links  provide  access  to  additional  information  at 
appropriate  stages  in  the  analysis. 

Mean  =  4.50,  SD  =  1.09,  95%  Cl  =  3.97  -  5.03,  n  =  19. 

Comments: 
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a.  Provides  information  discussion,  but  should  provide  step  by  step  to 
accomplish  Aircraft  Safe  for  Maintenance. 

b.  Some  lead  you  to  dead  ends  but  that  understandable. 

c.  Hypertext  links  are  a  good  method  to  get  at  additional  information.  Again 
a  lot  more  useful  links  could  be  made  given  enough  time  and  money. 

d.  User  should  be  able  to  access  any  screen  from  any  other  screen  (e.g.  a 
graphic  bar  on  screen  can  select  any  particular  screen  if  required). 

e.  Did  not  see. 

OmEML 

Question  20.  The  system  is  likely  to  be  more  acceptable  to  the  technician 
(end  user)  than  the  current  hardcopy  publications. 

Mean  *  3.84,  SD  *  0.99,  95%  Cl  =  3.37  -  4.30,  n  =  20, 

Comments; 

a.  A  full  up  system  may.  This  has  limited  application. 

b.  Past  experience  proves  that  the  user  is  not  always  exactly  sure  where  to 
look  for  information  (i.e.  the  IPB)  and  the  ability  to  pan  through  the 
hardcopy  is  needed. 

c.  Original  change  will  be  rejected,  but  through  use  program  will  be 
acceptable. 

d.  Old  habits  die  hard.  I  guess  in  the  long  term  it  will  be  cost  effective. 

e.  They  may  like  the  hypertext  links  a  lot  but  its  still  just  an  electronic 
version  of  the  paper. 
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f.  Technology  allows  us  to  integrate  systems  like  this  to  other  databases  and 
the  aircraft. 

g.  System  is  too  rigid. 

h.  Easy  to  use  format,  but  as  database  is  increased  will  the  amount  of 
information  help  or  overwhelm  the  technician?  Will  they  be  able  to  access 
everything  they  need? 

i.  Hard  copies  would  still  be  used  in  some  situations. 

j.  Good  for  training  purposes,  may  be  more  time  consuming  that  paper. 

k.  Computer  hardware  is  very  limited  in  my  workcenter. 

l.  Except  for  some  diflictdties,  like  pushing  button  on  mouse,  nothing 
happens. 

m.  I  think  they  will  still  need  some  information  -  especially  wiring  and 
schematics  in  hard  copy.  Don’t  throw  them  away  yet. 

n.  Everyone  will  resist  change. 

0.  Because  of  computer  downtime  to  reprogram  or  bad  weather,  it  could 
never  fully  replace  the  hardcopy. 

p.  I  believe  many  technicians  will  strongly  resist  change  (but  I  could  be 
wrong). 

Question  21.  The  system  is  suitable  for  users  with  different  levels  of 
knowledge/  experience. 

Mean  =  4.49,  SD  =  0.94,  95%  Cl  =  4.05  -  4.93,  n  =  20. 

Comments: 
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a.  No  levels  ofmaintenance  exist  that  I  saw,  i.e.,  expert  V8  novice.  Could  be 


implemented  fairly  easily  though. 

b.  However,  1  think  a  novice  might  get  lost  more  easily,  as  I  did. 

c.  Difficult  in  times. 

Question  22.  The  system  correctly  implements  the  fault  analysis  procedure 
as  presented  in  the  technical  order.  (Answer  Not  Applicable  if 
user  does  not  have  access  to  the  aircraft  publications.) 

Moan  =  4.75,  SD  =  1.26,  95%  Cl  =  2.75  -  6.75,  n  =  4. 

Comments: 

a.  System  didn’t  run  far  enough  to  evaluate.  Another  vote  for  hardcopy. 

b.  Program  is  an  on-screen  technical  order. 

Question  23.  The  system  provides  access  to  the  knowledge  necessary  for  the 

F-15E  Nose  Landing  Gear  Extraction  and  Retraction  Checkout 
Procedure.  (Answer  Not  Applicable  if  user  does  not  have  access 
to  the  aircraft  publications.) 

Mean  =  4.67,  SD  =  0.82,  95%  Cl  =  3.81  -  5.52,  n  =  6. 


Comments: 

Nil. 

Question  24.  The  system  is  more  efficient  than  using  conventional  technical 
order  publications.  (Answer  Not  Applicable  if  user  does  not 
have  access  to  the  aircraft  publications.) 

Mean  =  4.14,  SP  =  0.90,  95%  Cl  =  3.31  -  4.98,  n  =  7. 


Comments: 
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a.  More  direct.  Ignores  Not  Applicable  data. 

b.  Difficult  to  assess.  There  are  advantages  over  paper  TOs,  but  the 
difficulties  of  computers  have  not  been  experienced.  An  answer  better  after 
actual  flightline  test. 

c.  But,  I  expect  it  is. 

Question  25.  The  system  provides  expected  results. 

Mean  =  4.62,  SD  =  0.94,  95%  Cl  =  4.13  -  5.10,  n  =  17. 

Comments: 

a.  Followed  papercopy  Fault  Isolation. 

b.  Not  having  paper  TOs  its  tough  to  tell.  It  does  take  you  to  probable 
repairs  when  you  fail  a  test  at  the  end  of  a  tree  though. 

Question  26.  Overall,  the  system  is  useful. 

Mean  =  4.83,  SD  =  0.89,  95%  Cl  =  4.41  -  5.24,  n  =  20. 

Comments: 

a.  As  a  substitute  for  paper  TOs,  it  does  its  job  If  there  were  more  data  it 
might  be  easier  to  see  its  usefulness  and  full  capabilities. 

Question  27.  The  system  would  be  useful  in  a  workshop  environment. 

Mean  =  4.85,  SD  =  0.83,  95%  Cl  =  4.46  -  5.24,  n  =  20. 

Comments: 

a.  More  so  than  flightline.  Hardware  applications  need  exploration. 

b.  This  system  would  better  suit  the  in-shop  environment  than  the  flightline. 

c.  Wouldn’t  work  too  well  on  flightline,  since  it  needs  to  be  portable  and  not 
mouse  driven,  but  it  would  be  suitable  for  the  workshop. 
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d.  Consider  weather  factors,  availability  of  hardware,  and  possible  use  in  a 
wartime  scenario  -  maybe! 

Question  28.  The  system  would  be  useful  in  a  flightline  environment. 

Mean  =  4.20,  SD  =  1.37,  95%  Cl  =  3.56  -  4.84,  n  =  20. 

Comments: 

a.  Dependent  on  hardware. 

b.  If  technology  can  provide  on-screen  text  and  graphics  that  can  be  seen  in 
sunlight  and  a  battery  that  can  last  12  hours,  otherwise  it  would  be 
difficult  to  use  on  flightline. 

c.  Extremely  ruggedized  equipment  will  be  needed. 

d.  See  previous  comment. 

e.  If  all  available  on  Notebook  PC  would  allow  better  access  to  information. 

f.  Touch  screen  would  be  more  appropriate. 

g.  Only  if  compact  and  portable. 

h.  Not  ruggedized,  no  austere  capabilities. 

i.  If  could  be  made  portable  and  also  have  access  to  a  large  enough  monitor 
if  needed. 

j.  Consider  weather  factors,  availability  of  hardware,  and  possible  use  in  a 
wartime  scenario  -  maybe! 

Question  29.  The  system  would  be  useful  to  manage  technical  maintenance. 

Mean  =  4.82,  SD  =  0.97,  95%  Cl  =  4.36  -  5.27,  n  =  20. 

Comments: 

a.  Would  need  extension  of  the  program  and  mature  system. 
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b.  As  specialty  increases  and  shop  scope  decreases  the  technician  would  find 
the  computer  method  better  in  the  shop  environment. 

c.  Needs  work,  but  has  the  capability. 

Question  30.  The  system  could  be  used  to  manage  technical  publications. 

Mean  =  4.99,  SD  =  0.83,  95%  Cl  =  4.59  -  5.39,  n  =  19. 

Comments: 

a.  Needs  work,  but  has  the  capability. 

b.  The  system,  if  under  tight  configuration  management,  could  ensure  only 
accurate  information  is  in  use,  i.e.,  easy  audit. 

c.  With  proper  formatting. 

General  comments  received. 

a.  Scanned  graphics  have  poor  resolution  (common,  we  have  experienced  the 
same). 

b.  Callouts  need  to  be  selected,  and  I  assume  they  will  be. 

c.  Missing  data  for  a  lot  of  links. 

d.  List  of  available  graphics  is  OK,  but  why  not  make  a  menu  button  to  get 
at  them  or  only  display  those  for  the  steps  that  have  graphics. 

e.  Need  a  way  to  backup  in  a  procedure. 

f.  Need  a  way  to  change  the  answer  to  a  prompt. 

g.  If  a  fuller  database  existed,  I  assume  when  description,  operation,  TCTO, 
etc.  were  selected  that  only  information  for  the  particular  system  being 
worked  or  selected  would  be  displayed. 
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h.  Is  the  safety  button  warning  for  the  whole  task  or  for  a  particular  step? 
Kind  of  confusing. 

i.  General  maintenance  information  needs  to  be  structured  in  some  logical 
manner.  I  couldn’t  find  it.  Maybe  alphabetical  or  by  system? 

j.  Need  the  name  of  the  task  (test/repair)  displayed  on  the  titlebar  or 
somewhere. 

k.  The  name  of  the  link  on  the  right  side  title  bar  gets  truncated,  needs  to 
wrap  or  be  made  shorter. 

l.  Need  locator  graphics  for  a  lot  of  steps,  connectors,  doors,  etc. 

m.  When  a  repair  is  required,  why  not  take  them  right  to  the  repair  instead 
of  having  the  user  select  the  link  to  the  procedure? 

n.  The  links  for  support  data  give  the  user  no  idea  where  the  links  will  take 
them. 

0.  Why  keep  paper  references  in  electronic  format? 

p.  I  don’t  know  if  it  was  my  computer  or  what,  but  I  got  a  lot  of  Knowledge 
Fro  errors. 

q.  Why  stack  the  links  up  on  one  another. 

r.  Need  a  more  elegant  way  to  restart  from  the  beginning  rather  than  the 
start  of  the  checkout  procedure. 

s.  Backup?  How  do  I  get  back? 

t.  Soft  keys  -  contain  only  one  piece  of  information  and  it  is  static.  How  will 
organization/traversal  work  when  there  is  more  information  available? 
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u.  Graphics  -  how  do  they  tie  in  to  particular  procedures?  Or  do  you  just 
reference  whatever  you  think  you  need  to  see? 

V.  Procediire  format  was  confusing  -  it  didn’t  seem  to  have  much  continuity 
and  I  needed  titles  to  see  where  I  was  currently. 

w.  Hard  to  envision  an  overall  structure/procedure  hierarchy. 

X.  Wanted  to  get  back  to  original  sectional  aircraft  graphic. 

y.  Prompt  format  is  nice.  However,  would  it  be  helpful  to  know  what  the 
’correct’  response  is  on  a  checklist  like  this? 

z.  Waming/Note/Caution  format  is  good,  although  it  doesn’t  seem  obvious 
what  steps  they  are  applicable  to  ...  future  steps,  right? 

aa.  Why  is  support  equipment  stuck  in  the  middle  of  the  procedure?  Or  did  I 
get  sent  off  to  another  procedure  because  I  identified  a  fault?  Again,  it 
needs  titles  for  sets  of  procedural  steps  so  you  know  where  you  are. 

ab.  The  system  presents  current  documented  data  in  a  more  usable  medium. 

ac.  Locks  up  a  lot. 

ad.  Right  button  on  mouse  inoperable. 

ae.  Hard  to  go  back  to  screen  prior. 

af.  Hard  to  look  at  diagram  after  looking  at  procedures. 

ag.  Hard  to  get  to  troubleshooting  features. 

ah.  Can  be  a  good  program,  has  lot  of  potential. 

ai.  Like  to  show  and  explain  my  experience  to  someone  while  working  with  a 
computer. 

aj.  Great  start  on  an  overwhelming  project! 
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ak.  How  will  user  be  able  to  access  TO  information  of  other  than  primary 
system?  (Will  he  be  able  to  get  the  electric  or  hydraulic  TO  without  going 
back  to  the  shop  if  the  task  didn’t  list  it  as  a  requirement?) 

al.  Use  of  pen  notebook  in  low  light  situations? 

am.  TOs  do  take  a  lot  of  abuse.  This  is  an  expensive  bit  of  equipment,  and  111 
be  interested  to  see  how  you  are  going  to  make  it  ’mechanic  proof.’ 

an.  A  training  module  would  be  a  useful  addition  to  the  system.  A  system  that 
demonstrated  how  to  use  the  system  and  then  offered  practice  would  help 
introduce  new  users  to  the  system. 
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Clontarf  Beach,  Qld  4019 
Australia. 
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Vita 

Squadron  Leader  Glen  Steemson  was  bom  29  September  1961  in 
Bundaberg,  Queensland,  Australia.  He  enlisted  in  the  Royal  Australian  Air 
Force  in  1979  and  attended  the  Royal  Melbourne  Institute  of  Technology, 
Melbourne,  Australia.  In  1982  he  received  a  Bachelor  of  Engineering  majoring 
in  Electronic  Engineering,  and  was  commissioned  with  the  rank  of  Flying 
Officer. 

Postings  include  Engineering  Support  Flight  492  Squadron,  various  project 
management  positions  at  Headquarters  Logistics  Command,  and  Officer  in 
Charge  of  Avionics  Maintenance  for  the  P3C  Orion  aircraft  at  492  Squadron. 
He  was  posted  to  the  School  of  Systems  and  Logistics,  Air  Force  Institute  of 
Technology,  in  May  1991. 

Squadron  Leader  Steemson  is  married  with  one,  soon  to  be  two,  children. 

Permanent  Address;  8  F.E.  Walker  Street 

Bundaberg,  Qld  4670 
Australia. 
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